All of lore.kernel.org
 help / color / mirror / Atom feed
* btrfs check (not lowmem) and OOM-like hangs (4.17.6)
@ 2018-07-17 20:32 Marc MERLIN
  2018-07-17 20:59 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2018-07-17 20:32 UTC (permalink / raw)
  To: Su Yue; +Cc: Su Yue, quwenruo.btrfs, linux-btrfs

On Tue, Jul 17, 2018 at 10:50:32AM -0700, Marc MERLIN wrote:
> I got the following on 4.17.6 while running btrfs check --repair on an
> unmounted filesystem (not the lowmem version)
> 
> I understand that btrfs check is userland only, although it seems that
> it caused these FS hangs on a different filesystem (the trace of course
> does not provide info on which FS)
> 
> Any idea what happened here?
> I'm going to wait a few hours without running btrfs check to see if it
> happens again and then if running btrfs check will re-create this issue,
> but other suggestions (if any), are welcome:

Hi Qu, I know we were talking about this last week and then, btrfs check
just worked for me so I wasn't able to reproduce.
Now I'm able to reproduce again.

I tried again, it's definitely triggered by btrfs check --repair

I tried to capture what happens, and memory didn't dip to 0, but the system
got very slow and things started failing.
btrfs was never killed though while ssh was.
Is there a chance that maybe btrfs is in some kernel OOM exclude list?

Here is what I got when the system was not doing well (it took minutes to run):

             total       used       free     shared    buffers     cached
Mem:      32643788   32070952     572836          0     102160    4378772
-/+ buffers/cache:   27590020    5053768
Swap:     15616764     973596   14643168

gargamel:~# cat /proc/meminfo
MemTotal:       32643788 kB
MemFree:         2726276 kB
MemAvailable:    2502200 kB
Buffers:           12360 kB
Cached:          1676388 kB
SwapCached:     11048580 kB
Active:         16443004 kB
Inactive:       12010456 kB
Active(anon):   16287780 kB
Inactive(anon): 11651692 kB
Active(file):     155224 kB
Inactive(file):   358764 kB
Unevictable:        5776 kB
Mlocked:            5776 kB
SwapTotal:      15616764 kB
SwapFree:         294592 kB
Dirty:              3032 kB
Writeback:         76064 kB
AnonPages:      15723272 kB
Mapped:           612124 kB
Shmem:           1171032 kB
Slab:             399824 kB
SReclaimable:      84568 kB
SUnreclaim:       315256 kB
KernelStack:       20576 kB
PageTables:        94268 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    31938656 kB
Committed_AS:   37909452 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:     98304 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:          16384 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      355604 kB
DirectMap2M:    32897024 kB

and console:
[ 9184.345329] INFO: task zmtrigger.pl:9981 blocked for more than 120 seconds.
[ 9184.366258]       Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9184.385323] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9184.408803] zmtrigger.pl    D    0  9981   9804 0x20020080
[ 9184.425249] Call Trace:
[ 9184.432580]  ? __schedule+0x53e/0x59b
[ 9184.443551]  schedule+0x7f/0x98
[ 9184.452960]  io_schedule+0x16/0x38
[ 9184.463154]  wait_on_page_bit_common+0x10c/0x199
[ 9184.476996]  ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9184.493339]  shmem_getpage_gfp+0x2dd/0x975
[ 9184.506558]  shmem_fault+0x188/0x1c3
[ 9184.518199]  ? filemap_map_pages+0x6f/0x295
[ 9184.531680]  __do_fault+0x1d/0x6e
[ 9184.542505]  __handle_mm_fault+0x675/0xa61
[ 9184.555653]  ? list_move+0x21/0x3a
[ 9184.566737]  handle_mm_fault+0x11c/0x16b
[ 9184.579355]  __do_page_fault+0x324/0x41c
[ 9184.591996]  ? page_fault+0x8/0x30
[ 9184.603059]  page_fault+0x1e/0x30
[ 9184.613846] RIP: 0023:0xf7d2d022
[ 9184.624366] RSP: 002b:00000000ffeb9fe8 EFLAGS: 00010202
[ 9184.640868] RAX: 00000000f7eed000 RBX: 00000000567e6000 RCX: 0000000000000004
[ 9184.663095] RDX: 00000000587fecb0 RSI: 000000005876538c RDI: 0000000000000004
[ 9184.685308] RBP: 0000000058185160 R08: 0000000000000000 R09: 0000000000000000
[ 9184.707524] R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000000000
[ 9184.729757] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 9184.751988] INFO: task /usr/sbin/apach:11868 blocked for more than 120 seconds.
[ 9184.775106]       Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9184.795072] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9184.819423] /usr/sbin/apach D    0 11868  11311 0x20020080
[ 9184.836748] Call Trace:
[ 9184.844926]  ? __schedule+0x53e/0x59b
[ 9184.856811]  schedule+0x7f/0x98
[ 9184.867075]  io_schedule+0x16/0x38
[ 9184.878114]  wait_on_page_bit_common+0x10c/0x199
[ 9184.892807]  ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9184.909036]  shmem_getpage_gfp+0x2dd/0x975
[ 9184.922157]  shmem_fault+0x188/0x1c3
[ 9184.933667]  ? filemap_map_pages+0x6f/0x295
[ 9184.947504]  __do_fault+0x1d/0x6e
[ 9184.958234]  __handle_mm_fault+0x675/0xa61
[ 9184.971581]  handle_mm_fault+0x11c/0x16b
[ 9184.984114]  __do_page_fault+0x324/0x41c
[ 9184.996584]  ? page_fault+0x8/0x30
[ 9185.007462]  page_fault+0x1e/0x30
[ 9185.018085] RIP: 0023:0xf7aaa89e
[ 9185.028414] RSP: 002b:00000000ffb26f70 EFLAGS: 00210297
[ 9185.044686] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000000f7eb9ed8
[ 9185.066697] RDX: 00000000f3664820 RSI: 0000000056e8ad20 RDI: 00000000f7aaee40
[ 9185.088720] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[ 9185.110713] R10: 0000000000000000 R11: 0000000000200282 R12: 0000000000000000
[ 9185.132677] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 9305.182796] INFO: task atop:6459 blocked for more than 120 seconds.
[ 9305.202538]       Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9305.222445] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9305.246536] atop            D    0  6459      1 0x20020080
[ 9305.263597] Call Trace:
[ 9305.271548]  ? __schedule+0x53e/0x59b
[ 9305.283242]  schedule+0x7f/0x98
[ 9305.293336]  io_schedule+0x16/0x38
[ 9305.304602]  wait_on_page_bit_common+0x10c/0x199
[ 9305.319074]  ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9305.335182]  __lock_page_or_retry+0x6c/0x75
[ 9305.348811]  do_swap_page+0x21f/0x511
[ 9305.360470]  __handle_mm_fault+0x9c2/0xa61
[ 9305.373441]  ? zone_statistics+0x4c/0x60
[ 9305.385854]  handle_mm_fault+0x11c/0x16b
[ 9305.398797]  __get_user_pages+0x364/0x505
[ 9305.411513]  get_user_pages_remote+0x8e/0x18d
[ 9305.425209]  __access_remote_vm+0x8c/0x1b4
[ 9305.438135]  proc_pid_cmdline_read+0x13b/0x3dd
[ 9305.452150]  ? do_filp_open+0x99/0xc6
[ 9305.463786]  ? __vfs_read+0x33/0xcc
[ 9305.474901]  __vfs_read+0x33/0xcc
[ 9305.485476]  ? current_kernel_time64+0x10/0x36
[ 9305.499916]  vfs_read+0x97/0xcc
[ 9305.510013]  ksys_read+0x5c/0x9f
[ 9305.520301]  do_fast_syscall_32+0xaf/0xf3
[ 9305.533157]  entry_SYSENTER_compat+0x7f/0x91
[ 9305.546619] INFO: task zmtrigger.pl:9981 blocked for more than 120 seconds.
[ 9305.568354]       Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9305.588237] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9305.612340] zmtrigger.pl    D    0  9981   9804 0x20020080
[ 9305.629607] Call Trace:
[ 9305.637714]  ? __schedule+0x53e/0x59b
[ 9305.649350]  schedule+0x7f/0x98
[ 9305.659441]  io_schedule+0x16/0x38
[ 9305.670283]  wait_on_page_bit_common+0x10c/0x199
[ 9305.684771]  ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9305.700839]  shmem_getpage_gfp+0x2dd/0x975
[ 9305.713854]  shmem_fault+0x188/0x1c3
[ 9305.725230]  ? filemap_map_pages+0x6f/0x295
[ 9305.738416]  __do_fault+0x1d/0x6e
[ 9305.749000]  __handle_mm_fault+0x675/0xa61
[ 9305.761923]  ? list_move+0x21/0x3a
[ 9305.772857]  handle_mm_fault+0x11c/0x16b
[ 9305.785262]  __do_page_fault+0x324/0x41c
[ 9305.797733]  ? page_fault+0x8/0x30
[ 9305.808581]  page_fault+0x1e/0x30
[ 9305.819429] RIP: 0023:0xf7d2d022
[ 9305.829797] RSP: 002b:00000000ffeb9fe8 EFLAGS: 00010202
[ 9305.846085] RAX: 00000000f7eed000 RBX: 00000000567e6000 RCX: 0000000000000004
[ 9305.868212] RDX: 00000000587fecb0 RSI: 000000005876538c RDI: 0000000000000004
[ 9305.890279] RBP: 0000000058185160 R08: 0000000000000000 R09: 0000000000000000
[ 9305.912266] R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000000000
[ 9305.934291] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 9305.956295] INFO: task /usr/sbin/apach:11868 blocked for more than 120 seconds.
[ 9305.978927]       Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9305.998652] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9306.022742] /usr/sbin/apach D    0 11868  11311 0x20020080
[ 9306.040041] Call Trace:
[ 9306.048029]  ? __schedule+0x53e/0x59b
[ 9306.059703]  schedule+0x7f/0x98
[ 9306.069772]  io_schedule+0x16/0x38
[ 9306.080615]  wait_on_page_bit_common+0x10c/0x199
[ 9306.095146]  ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9306.111423]  shmem_getpage_gfp+0x2dd/0x975
[ 9306.124384]  shmem_fault+0x188/0x1c3
[ 9306.135746]  ? filemap_map_pages+0x6f/0x295
[ 9306.148943]  __do_fault+0x1d/0x6e
[ 9306.159540]  __handle_mm_fault+0x675/0xa61
[ 9306.172531]  handle_mm_fault+0x11c/0x16b
[ 9306.184947]  __do_page_fault+0x324/0x41c
[ 9306.197544]  ? page_fault+0x8/0x30
[ 9306.208405]  page_fault+0x1e/0x30
[ 9306.219016] RIP: 0023:0xf7aaa89e
[ 9306.229524] RSP: 002b:00000000ffb26f70 EFLAGS: 00210297
[ 9306.245994] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000000f7eb9ed8
[ 9306.268285] RDX: 00000000f3664820 RSI: 0000000056e8ad20 RDI: 00000000f7aaee40
[ 9306.290641] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[ 9306.313028] R10: 0000000000000000 R11: 0000000000200282 R12: 0000000000000000
[ 9306.335441] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 9306.357849] INFO: task pgrep:5570 blocked for more than 120 seconds.
[ 9306.377918]       Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9306.397989] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9306.422282] pgrep           D    0  5570   5567 0x20020080
[ 9306.439795] Call Trace:
[ 9306.447754]  ? __schedule+0x53e/0x59b
[ 9306.459419]  schedule+0x7f/0x98
[ 9306.469460]  io_schedule+0x16/0x38
[ 9306.480275]  wait_on_page_bit_common+0x10c/0x199
[ 9306.494738]  ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9306.510783]  __lock_page_or_retry+0x6c/0x75
[ 9306.523970]  do_swap_page+0x21f/0x511
[ 9306.535593]  ? btrfs_bio_counter_sub+0x1e/0x4c
[ 9306.549560]  __handle_mm_fault+0x9c2/0xa61
[ 9306.562483]  ? ktime_get+0x31/0x41
[ 9306.573324]  handle_mm_fault+0x11c/0x16b
[ 9306.585723]  __get_user_pages+0x364/0x505
[ 9306.598377]  get_user_pages_remote+0x8e/0x18d
[ 9306.612083]  __access_remote_vm+0x8c/0x1b4
[ 9306.625009]  proc_pid_cmdline_read+0x13b/0x3dd
[ 9306.638977]  ? do_filp_open+0x99/0xc6
[ 9306.650599]  ? __vfs_read+0x33/0xcc
[ 9306.661690]  __vfs_read+0x33/0xcc
[ 9306.672257]  ? current_kernel_time64+0x10/0x36
[ 9306.686182]  vfs_read+0x97/0xcc
[ 9306.696363]  ksys_read+0x5c/0x9f
[ 9306.706638]  do_fast_syscall_32+0xaf/0xf3
[ 9306.719293]  entry_SYSENTER_compat+0x7f/0x91
[ 9306.732717] INFO: task pgrep:5579 blocked for more than 120 seconds.
[ 9306.752547]       Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9306.772223] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9306.796319] pgrep           D    0  5579   4327 0x20020080
[ 9306.813577] Call Trace:
[ 9306.821562]  ? __schedule+0x53e/0x59b
[ 9306.833180]  schedule+0x7f/0x98
[ 9306.843275]  io_schedule+0x16/0x38
[ 9306.854164]  wait_on_page_bit_common+0x10c/0x199
[ 9306.868651]  ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9306.884730]  __lock_page_or_retry+0x6c/0x75
[ 9306.897942]  do_swap_page+0x21f/0x511
[ 9306.909566]  ? aac_scsi_cmd+0x455/0x117b [aacraid]
[ 9306.924630]  __handle_mm_fault+0x9c2/0xa61
[ 9306.937594]  ? ktime_get+0x31/0x41
[ 9306.948478]  handle_mm_fault+0x11c/0x16b
[ 9306.961081]  __get_user_pages+0x364/0x505
[ 9306.973748]  get_user_pages_remote+0x8e/0x18d
[ 9306.987598]  __access_remote_vm+0x8c/0x1b4
[ 9307.000744]  proc_pid_cmdline_read+0x13b/0x3dd
[ 9307.014868]  ? do_filp_open+0x99/0xc6
[ 9307.026676]  ? __vfs_read+0x33/0xcc
[ 9307.037927]  __vfs_read+0x33/0xcc
[ 9307.048670]  ? current_kernel_time64+0x10/0x36
[ 9307.062789]  vfs_read+0x97/0xcc
[ 9307.072860]  ksys_read+0x5c/0x9f
[ 9307.083171]  do_fast_syscall_32+0xaf/0xf3
[ 9307.095810]  entry_SYSENTER_compat+0x7f/0x91
[ 9428.060462] INFO: task atop:6459 blocked for more than 120 seconds.
[ 9428.080112]       Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9428.099816] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9428.123975] atop            D    0  6459      1 0x20020080
[ 9428.141515] Call Trace:
[ 9428.149507]  ? __schedule+0x53e/0x59b
[ 9428.161247]  schedule+0x7f/0x98
[ 9428.171320]  io_schedule+0x16/0x38
[ 9428.182181]  wait_on_page_bit_common+0x10c/0x199
[ 9428.196674]  ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9428.212767]  __lock_page_or_retry+0x6c/0x75
[ 9428.225983]  do_swap_page+0x21f/0x511
[ 9428.237625]  __handle_mm_fault+0x9c2/0xa61
[ 9428.250579]  ? zone_statistics+0x4c/0x60
[ 9428.262977]  handle_mm_fault+0x11c/0x16b
[ 9428.275429]  __get_user_pages+0x364/0x505
[ 9428.288117]  get_user_pages_remote+0x8e/0x18d
[ 9428.301873]  __access_remote_vm+0x8c/0x1b4
[ 9428.314854]  proc_pid_cmdline_read+0x13b/0x3dd
[ 9428.328897]  ? do_filp_open+0x99/0xc6
[ 9428.340536]  ? __vfs_read+0x33/0xcc
[ 9428.351626]  __vfs_read+0x33/0xcc
[ 9428.362181]  ? current_kernel_time64+0x10/0x36
[ 9428.376107]  vfs_read+0x97/0xcc
[ 9428.386112]  ksys_read+0x5c/0x9f
[ 9428.396406]  do_fast_syscall_32+0xaf/0xf3
[ 9428.409029]  entry_SYSENTER_compat+0x7f/0x91
[ 9428.422465] INFO: task zmtrigger.pl:9981 blocked for more than 120 seconds.
[ 9428.443947]       Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9428.463635] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9428.487740] zmtrigger.pl    D    0  9981   9804 0x20020080
[ 9428.504827] Call Trace:
[ 9428.512804]  ? __schedule+0x53e/0x59b
[ 9428.524426]  schedule+0x7f/0x98
[ 9428.534486]  io_schedule+0x16/0x38
[ 9428.545325]  wait_on_page_bit_common+0x10c/0x199
[ 9428.559813]  ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9428.575860]  shmem_getpage_gfp+0x2dd/0x975
[ 9428.588802]  shmem_fault+0x188/0x1c3
[ 9428.600161]  ? filemap_map_pages+0x6f/0x295
[ 9428.613348]  __do_fault+0x1d/0x6e
[ 9428.623960]  __handle_mm_fault+0x675/0xa61
[ 9428.636934]  ? list_move+0x21/0x3a
[ 9428.647968]  handle_mm_fault+0x11c/0x16b
[ 9428.660517]  __do_page_fault+0x324/0x41c
[ 9428.672990]  ? page_fault+0x8/0x30
[ 9428.683846]  page_fault+0x1e/0x30
[ 9428.694437] RIP: 0023:0xf7d2d022
[ 9428.704744] RSP: 002b:00000000ffeb9fe8 EFLAGS: 00010202
[ 9428.721025] RAX: 00000000f7eed000 RBX: 00000000567e6000 RCX: 0000000000000004
[ 9428.743062] RDX: 00000000587fecb0 RSI: 000000005876538c RDI: 0000000000000004
[ 9428.765091] RBP: 0000000058185160 R08: 0000000000000000 R09: 0000000000000000
[ 9428.787241] R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000000000
[ 9428.809265] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 9428.831501] INFO: task /usr/sbin/apach:11868 blocked for more than 120 seconds.
[ 9428.854070]       Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9428.873815] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9428.897937] /usr/sbin/apach D    0 11868  11311 0x20020080
[ 9428.915023] Call Trace:
[ 9428.923002]  ? __schedule+0x53e/0x59b
[ 9428.934625]  schedule+0x7f/0x98
[ 9428.944687]  io_schedule+0x16/0x38
[ 9428.955525]  wait_on_page_bit_common+0x10c/0x199
[ 9428.970014]  ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9428.986074]  shmem_getpage_gfp+0x2dd/0x975
[ 9428.999006]  shmem_fault+0x188/0x1c3
[ 9429.010366]  ? filemap_map_pages+0x6f/0x295
[ 9429.023551]  __do_fault+0x1d/0x6e
[ 9429.034140]  __handle_mm_fault+0x675/0xa61
[ 9429.047056]  handle_mm_fault+0x11c/0x16b
[ 9429.059461]  __do_page_fault+0x324/0x41c
[ 9429.071857]  ? page_fault+0x8/0x30
[ 9429.082703]  page_fault+0x1e/0x30
[ 9429.093276] RIP: 0023:0xf7aaa89e
[ 9429.103591] RSP: 002b:00000000ffb26f70 EFLAGS: 00210297
[ 9429.119892] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000000f7eb9ed8

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6)
  2018-07-17 20:32 btrfs check (not lowmem) and OOM-like hangs (4.17.6) Marc MERLIN
@ 2018-07-17 20:59 ` Marc MERLIN
  2018-07-18  0:05   ` Qu Wenruo
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2018-07-17 20:59 UTC (permalink / raw)
  To: Su Yue; +Cc: Su Yue, quwenruo.btrfs, linux-btrfs

Ok, I did more testing. Qu is right that btrfs check does not crash the kernel.
It just takes all the memory until linux hangs everywhere, and somehow (no idea why) 
the OOM killer never triggers.
Details below:

On Tue, Jul 17, 2018 at 01:32:57PM -0700, Marc MERLIN wrote:
> Here is what I got when the system was not doing well (it took minutes to run):
> 
>              total       used       free     shared    buffers     cached
> Mem:      32643788   32070952     572836          0     102160    4378772
> -/+ buffers/cache:   27590020    5053768
> Swap:     15616764     973596   14643168

ok, the reason it was not that close to 0 was due to /dev/shm it seems.
I cleared that, and now I can get it to go to near 0 again.
I'm wrong about the system being fully crashed, it's not, it's just very
close to being hung.
I can type killall -9 btrfs in the serial console and wait a few minutes.
The system eventually recovers, but it's impossible to fix anything via ssh 
apparently because networking does not get to run when I'm in this state.

I'm not sure why my system reproduces this easy while Qu's system does not, 
but Qu was right that the kernel is not dead and that it's merely a problem of userspace
taking all the RAM and somehow not being killed by OOM

I checked the PID and don't see why it's not being killed:
gargamel:/proc/31006# grep . oom*
oom_adj:0
oom_score:221   << this increases a lot, but OOM never kills it
oom_score_adj:0

I have these variables:
/proc/sys/vm/oom_dump_tasks:1
/proc/sys/vm/oom_kill_allocating_task:0
/proc/sys/vm/overcommit_kbytes:0
/proc/sys/vm/overcommit_memory:0
/proc/sys/vm/overcommit_ratio:50  << is this bad (seems default)

Here is my system when it virtually died:
ER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49   1:35 ./btrfs check /dev/mapper/dshelf2

             total       used       free     shared    buffers     cached
Mem:      32643788   32180100     463688          0      44664     119508
-/+ buffers/cache:   32015928     627860
Swap:     15616764     443676   15173088

MemTotal:       32643788 kB
MemFree:          463440 kB
MemAvailable:      44864 kB
Buffers:           44664 kB
Cached:           120360 kB
SwapCached:        87064 kB
Active:         30381404 kB
Inactive:         585952 kB
Active(anon):   30334696 kB
Inactive(anon):   474624 kB
Active(file):      46708 kB
Inactive(file):   111328 kB
Unevictable:        5616 kB
Mlocked:            5616 kB
SwapTotal:      15616764 kB
SwapFree:       15173088 kB
Dirty:              1636 kB
Writeback:             4 kB
AnonPages:      30734240 kB
Mapped:            67236 kB
Shmem:              3036 kB
Slab:             267884 kB
SReclaimable:      51528 kB
SUnreclaim:       216356 kB
KernelStack:       10144 kB
PageTables:        69284 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    31938656 kB
Committed_AS:   32865492 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:          16384 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      560404 kB
DirectMap2M:    32692224 kB


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6)
  2018-07-17 20:59 ` Marc MERLIN
@ 2018-07-18  0:05   ` Qu Wenruo
  2018-07-18  0:24     ` Marc MERLIN
  2018-07-18 19:42     ` btrfs check (not lowmem) and OOM-like hangs (4.17.6) Andrei Borzenkov
  0 siblings, 2 replies; 479+ messages in thread
From: Qu Wenruo @ 2018-07-18  0:05 UTC (permalink / raw)
  To: Marc MERLIN, Su Yue; +Cc: Su Yue, linux-btrfs



On 2018年07月18日 04:59, Marc MERLIN wrote:
> Ok, I did more testing. Qu is right that btrfs check does not crash the kernel.
> It just takes all the memory until linux hangs everywhere, and somehow (no idea why) 
> the OOM killer never triggers.

No OOM triggers? That's a little strange.
Maybe it's related to how kernel handles memory over-commit?

And for the hang, I think it's related to some memory allocation failure
and error handler just didn't handle it well, so it's causing deadlock
for certain page.

ENOMEM handling is pretty common but hardly verified, so it's not that
strange, but we must locate the problem.

> Details below:
> 
> On Tue, Jul 17, 2018 at 01:32:57PM -0700, Marc MERLIN wrote:
>> Here is what I got when the system was not doing well (it took minutes to run):
>>
>>              total       used       free     shared    buffers     cached
>> Mem:      32643788   32070952     572836          0     102160    4378772
>> -/+ buffers/cache:   27590020    5053768
>> Swap:     15616764     973596   14643168
> 
> ok, the reason it was not that close to 0 was due to /dev/shm it seems.
> I cleared that, and now I can get it to go to near 0 again.
> I'm wrong about the system being fully crashed, it's not, it's just very
> close to being hung.
> I can type killall -9 btrfs in the serial console and wait a few minutes.
> The system eventually recovers, but it's impossible to fix anything via ssh 
> apparently because networking does not get to run when I'm in this state.
> 
> I'm not sure why my system reproduces this easy while Qu's system does not, 
> but Qu was right that the kernel is not dead and that it's merely a problem of userspace
> taking all the RAM and somehow not being killed by OOM

In my system, at least I'm not using btrfs as root fs, and for the
memory eating program I normally ensure it's eating all the memory +
swap, so OOM killer is always triggered, maybe that's the cause.

So in your case, maybe it's btrfs not really taking up all memory, thus
OOM killer not triggered.

> 
> I checked the PID and don't see why it's not being killed:
> gargamel:/proc/31006# grep . oom*
> oom_adj:0
> oom_score:221   << this increases a lot, but OOM never kills it
> oom_score_adj:0
> 
> I have these variables:
> /proc/sys/vm/oom_dump_tasks:1
> /proc/sys/vm/oom_kill_allocating_task:0
> /proc/sys/vm/overcommit_kbytes:0
> /proc/sys/vm/overcommit_memory:0
> /proc/sys/vm/overcommit_ratio:50  << is this bad (seems default)

Any kernel dmesg about OOM killer triggered?

> 
> Here is my system when it virtually died:
> ER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> root     31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49   1:35 ./btrfs check /dev/mapper/dshelf2
> 
>              total       used       free     shared    buffers     cached
> Mem:      32643788   32180100     463688          0      44664     119508
> -/+ buffers/cache:   32015928     627860
> Swap:     15616764     443676   15173088

For swap, it looks like only some other program's memory is swapped out,
not btrfs'.

And unfortunately, I'm not so familiar with OOM/MM code outside of
filesystem.
Any help from other experienced developers would definitely help to
solve why memory of 'btrfs check' is not swapped out or why OOM killer
is not triggered.

Thanks,
Qu

> 
> MemTotal:       32643788 kB
> MemFree:          463440 kB
> MemAvailable:      44864 kB
> Buffers:           44664 kB
> Cached:           120360 kB
> SwapCached:        87064 kB
> Active:         30381404 kB
> Inactive:         585952 kB
> Active(anon):   30334696 kB
> Inactive(anon):   474624 kB
> Active(file):      46708 kB
> Inactive(file):   111328 kB
> Unevictable:        5616 kB
> Mlocked:            5616 kB
> SwapTotal:      15616764 kB
> SwapFree:       15173088 kB
> Dirty:              1636 kB
> Writeback:             4 kB
> AnonPages:      30734240 kB
> Mapped:            67236 kB
> Shmem:              3036 kB
> Slab:             267884 kB
> SReclaimable:      51528 kB
> SUnreclaim:       216356 kB
> KernelStack:       10144 kB
> PageTables:        69284 kB
> NFS_Unstable:          0 kB
> Bounce:                0 kB
> WritebackTmp:          0 kB
> CommitLimit:    31938656 kB
> Committed_AS:   32865492 kB
> VmallocTotal:   34359738367 kB
> VmallocUsed:           0 kB
> VmallocChunk:          0 kB
> HardwareCorrupted:     0 kB
> AnonHugePages:         0 kB
> ShmemHugePages:        0 kB
> ShmemPmdMapped:        0 kB
> CmaTotal:          16384 kB
> CmaFree:               0 kB
> HugePages_Total:       0
> HugePages_Free:        0
> HugePages_Rsvd:        0
> HugePages_Surp:        0
> Hugepagesize:       2048 kB
> Hugetlb:               0 kB
> DirectMap4k:      560404 kB
> DirectMap2M:    32692224 kB
> 
> 

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6)
  2018-07-18  0:05   ` Qu Wenruo
@ 2018-07-18  0:24     ` Marc MERLIN
  2018-07-19  9:22       ` Qu Wenruo
                         ` (2 more replies)
  2018-07-18 19:42     ` btrfs check (not lowmem) and OOM-like hangs (4.17.6) Andrei Borzenkov
  1 sibling, 3 replies; 479+ messages in thread
From: Marc MERLIN @ 2018-07-18  0:24 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: linux-btrfs

On Wed, Jul 18, 2018 at 08:05:51AM +0800, Qu Wenruo wrote:
> No OOM triggers? That's a little strange.
> Maybe it's related to how kernel handles memory over-commit?
 
Yes, I think you are correct.

> And for the hang, I think it's related to some memory allocation failure
> and error handler just didn't handle it well, so it's causing deadlock
> for certain page.

That indeed matches what I'm seeing.

> ENOMEM handling is pretty common but hardly verified, so it's not that
> strange, but we must locate the problem.

I seem to be getting deadlocks in the kernel, so I'm hoping that at least
it's checked there, but maybe not?

> In my system, at least I'm not using btrfs as root fs, and for the
> memory eating program I normally ensure it's eating all the memory +
> swap, so OOM killer is always triggered, maybe that's the cause.
> 
> So in your case, maybe it's btrfs not really taking up all memory, thus
> OOM killer not triggered.

Correct, the swap is not used.

> Any kernel dmesg about OOM killer triggered?
 
Nothing at all. It never gets triggered.

> > Here is my system when it virtually died:
> > ER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> > root     31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49   1:35 ./btrfs check /dev/mapper/dshelf2

See how btrs was taking 29GB in that ps output (that's before it takes
everything and I can't even type ps anymore)
Note that VSZ is almost equal to RSS. Nothing gets swapped.

Then see free output:

> >              total       used       free     shared    buffers     cached
> > Mem:      32643788   32180100     463688          0      44664     119508
> > -/+ buffers/cache:   32015928     627860
> > Swap:     15616764     443676   15173088
> 
> For swap, it looks like only some other program's memory is swapped out,
> not btrfs'.

That's exactly correct. btrfs check never goes to swap, I'm not sure why,
and because there is virtual memory free, maybe that's why OOM does not
trigger?
So I guess I can probably "fix" my problem by removing swap, but ultimately
it would be useful to know why memory taken by btrfs check does not end up
in swap.

> And unfortunately, I'm not so familiar with OOM/MM code outside of
> filesystem.
> Any help from other experienced developers would definitely help to
> solve why memory of 'btrfs check' is not swapped out or why OOM killer
> is not triggered.

Do you have someone from linux-vm you might be able to ask, or should we Cc
this thread there?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6)
  2018-07-18  0:05   ` Qu Wenruo
  2018-07-18  0:24     ` Marc MERLIN
@ 2018-07-18 19:42     ` Andrei Borzenkov
  2018-07-18 21:56       ` Marc MERLIN
  1 sibling, 1 reply; 479+ messages in thread
From: Andrei Borzenkov @ 2018-07-18 19:42 UTC (permalink / raw)
  To: Qu Wenruo, Marc MERLIN, Su Yue; +Cc: Su Yue, linux-btrfs

18.07.2018 03:05, Qu Wenruo пишет:
> 
> 
> On 2018年07月18日 04:59, Marc MERLIN wrote:
>> Ok, I did more testing. Qu is right that btrfs check does not crash the kernel.
>> It just takes all the memory until linux hangs everywhere, and somehow (no idea why) 
>> the OOM killer never triggers.
> 
> No OOM triggers? That's a little strange.
> Maybe it's related to how kernel handles memory over-commit?
> 
> And for the hang, I think it's related to some memory allocation failure
> and error handler just didn't handle it well, so it's causing deadlock
> for certain page.
> 
> ENOMEM handling is pretty common but hardly verified, so it's not that
> strange, but we must locate the problem.
> 
>> Details below:
>>
>> On Tue, Jul 17, 2018 at 01:32:57PM -0700, Marc MERLIN wrote:
>>> Here is what I got when the system was not doing well (it took minutes to run):
>>>
>>>              total       used       free     shared    buffers     cached
>>> Mem:      32643788   32070952     572836          0     102160    4378772
>>> -/+ buffers/cache:   27590020    5053768
>>> Swap:     15616764     973596   14643168
>>
>> ok, the reason it was not that close to 0 was due to /dev/shm it seems.
>> I cleared that, and now I can get it to go to near 0 again.
>> I'm wrong about the system being fully crashed, it's not, it's just very
>> close to being hung.
>> I can type killall -9 btrfs in the serial console and wait a few minutes.
>> The system eventually recovers, but it's impossible to fix anything via ssh 
>> apparently because networking does not get to run when I'm in this state.
>>
>> I'm not sure why my system reproduces this easy while Qu's system does not, 
>> but Qu was right that the kernel is not dead and that it's merely a problem of userspace
>> taking all the RAM and somehow not being killed by OOM
> 
> In my system, at least I'm not using btrfs as root fs, and for the
> memory eating program I normally ensure it's eating all the memory +
> swap, so OOM killer is always triggered, maybe that's the cause.
> 
> So in your case, maybe it's btrfs not really taking up all memory, thus
> OOM killer not triggered.
> 
>>
>> I checked the PID and don't see why it's not being killed:
>> gargamel:/proc/31006# grep . oom*
>> oom_adj:0
>> oom_score:221   << this increases a lot, but OOM never kills it
>> oom_score_adj:0
>>
>> I have these variables:
>> /proc/sys/vm/oom_dump_tasks:1
>> /proc/sys/vm/oom_kill_allocating_task:0
>> /proc/sys/vm/overcommit_kbytes:0
>> /proc/sys/vm/overcommit_memory:0
>> /proc/sys/vm/overcommit_ratio:50  << is this bad (seems default)
> 
> Any kernel dmesg about OOM killer triggered?
> 
>>
>> Here is my system when it virtually died:
>> ER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
>> root     31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49   1:35 ./btrfs check /dev/mapper/dshelf2
>>
>>              total       used       free     shared    buffers     cached
>> Mem:      32643788   32180100     463688          0      44664     119508
>> -/+ buffers/cache:   32015928     627860
>> Swap:     15616764     443676   15173088
> 
> For swap, it looks like only some other program's memory is swapped out,
> not btrfs'.
> 
> And unfortunately, I'm not so familiar with OOM/MM code outside of
> filesystem.
> Any help from other experienced developers would definitely help to
> solve why memory of 'btrfs check' is not swapped out or why OOM killer
> is not triggered.
> 

Almost all used memory is marked as "active" and active pages are not
swapped. Page is active if it was accessed recently. Is it possible that
btrfs logic does frequent scans across all allocated memory?



> Thanks,
> Qu
> 
>>
>> MemTotal:       32643788 kB
>> MemFree:          463440 kB
>> MemAvailable:      44864 kB
>> Buffers:           44664 kB
>> Cached:           120360 kB
>> SwapCached:        87064 kB
>> Active:         30381404 kB
>> Inactive:         585952 kB
>> Active(anon):   30334696 kB
>> Inactive(anon):   474624 kB
>> Active(file):      46708 kB
>> Inactive(file):   111328 kB
>> Unevictable:        5616 kB
>> Mlocked:            5616 kB
>> SwapTotal:      15616764 kB
>> SwapFree:       15173088 kB
>> Dirty:              1636 kB
>> Writeback:             4 kB
>> AnonPages:      30734240 kB
>> Mapped:            67236 kB
>> Shmem:              3036 kB
>> Slab:             267884 kB
>> SReclaimable:      51528 kB
>> SUnreclaim:       216356 kB
>> KernelStack:       10144 kB
>> PageTables:        69284 kB
>> NFS_Unstable:          0 kB
>> Bounce:                0 kB
>> WritebackTmp:          0 kB
>> CommitLimit:    31938656 kB
>> Committed_AS:   32865492 kB
>> VmallocTotal:   34359738367 kB
>> VmallocUsed:           0 kB
>> VmallocChunk:          0 kB
>> HardwareCorrupted:     0 kB
>> AnonHugePages:         0 kB
>> ShmemHugePages:        0 kB
>> ShmemPmdMapped:        0 kB
>> CmaTotal:          16384 kB
>> CmaFree:               0 kB
>> HugePages_Total:       0
>> HugePages_Free:        0
>> HugePages_Rsvd:        0
>> HugePages_Surp:        0
>> Hugepagesize:       2048 kB
>> Hugetlb:               0 kB
>> DirectMap4k:      560404 kB
>> DirectMap2M:    32692224 kB
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6)
  2018-07-18 19:42     ` btrfs check (not lowmem) and OOM-like hangs (4.17.6) Andrei Borzenkov
@ 2018-07-18 21:56       ` Marc MERLIN
  0 siblings, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2018-07-18 21:56 UTC (permalink / raw)
  To: Andrei Borzenkov; +Cc: Qu Wenruo, Su Yue, Su Yue, linux-btrfs

On Wed, Jul 18, 2018 at 10:42:21PM +0300, Andrei Borzenkov wrote:
> > Any help from other experienced developers would definitely help to
> > solve why memory of 'btrfs check' is not swapped out or why OOM killer
> > is not triggered.
> 
> Almost all used memory is marked as "active" and active pages are not
> swapped. Page is active if it was accessed recently. Is it possible that
> btrfs logic does frequent scans across all allocated memory?
> >>
> >> Active:         30381404 kB
> >> Inactive:         585952 kB

That is a very good find.

Yes, the linux kernel VM may be smart enough not to swap pages that got used
recently and when btrfs slurps all the extents to cross check everything, I
think it does cross reference them all many times.
This is why it can run in a few hours when btrfs check lowmem requires days
to run in a similar situation.

I'm not sure if there is a good way around this, but it's good to know that
btrfs repair can effectively abuse the linux VM in a way that it'll take
everything down without OOM having a chance to trigger.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6)
  2018-07-18  0:24     ` Marc MERLIN
@ 2018-07-19  9:22       ` Qu Wenruo
  2019-10-18  2:56       ` 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 Marc MERLIN
  2020-05-24 21:30       ` 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' Marc MERLIN
  2 siblings, 0 replies; 479+ messages in thread
From: Qu Wenruo @ 2018-07-19  9:22 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 3526 bytes --]



On 2018年07月18日 08:24, Marc MERLIN wrote:
> On Wed, Jul 18, 2018 at 08:05:51AM +0800, Qu Wenruo wrote:
>> No OOM triggers? That's a little strange.
>> Maybe it's related to how kernel handles memory over-commit?
>  
> Yes, I think you are correct.
> 
>> And for the hang, I think it's related to some memory allocation failure
>> and error handler just didn't handle it well, so it's causing deadlock
>> for certain page.
> 
> That indeed matches what I'm seeing.
> 
>> ENOMEM handling is pretty common but hardly verified, so it's not that
>> strange, but we must locate the problem.
> 
> I seem to be getting deadlocks in the kernel, so I'm hoping that at least
> it's checked there, but maybe not?
> 
>> In my system, at least I'm not using btrfs as root fs, and for the
>> memory eating program I normally ensure it's eating all the memory +
>> swap, so OOM killer is always triggered, maybe that's the cause.
>>
>> So in your case, maybe it's btrfs not really taking up all memory, thus
>> OOM killer not triggered.
> 
> Correct, the swap is not used.
> 
>> Any kernel dmesg about OOM killer triggered?
>  
> Nothing at all. It never gets triggered.
> 
>>> Here is my system when it virtually died:
>>> ER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
>>> root     31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49   1:35 ./btrfs check /dev/mapper/dshelf2
> 
> See how btrs was taking 29GB in that ps output (that's before it takes
> everything and I can't even type ps anymore)
> Note that VSZ is almost equal to RSS. Nothing gets swapped.
> 
> Then see free output:
> 
>>>              total       used       free     shared    buffers     cached
>>> Mem:      32643788   32180100     463688          0      44664     119508
>>> -/+ buffers/cache:   32015928     627860
>>> Swap:     15616764     443676   15173088
>>
>> For swap, it looks like only some other program's memory is swapped out,
>> not btrfs'.
> 
> That's exactly correct. btrfs check never goes to swap, I'm not sure why,
> and because there is virtual memory free, maybe that's why OOM does not
> trigger?
> So I guess I can probably "fix" my problem by removing swap, but ultimately
> it would be useful to know why memory taken by btrfs check does not end up
> in swap.
> 
>> And unfortunately, I'm not so familiar with OOM/MM code outside of
>> filesystem.
>> Any help from other experienced developers would definitely help to
>> solve why memory of 'btrfs check' is not swapped out or why OOM killer
>> is not triggered.
> 
> Do you have someone from linux-vm you might be able to ask, or should we Cc
> this thread there?

Michal Hocho gives me a brief session about this.
Which is super helpful in this case, thank you Michal!

Firstly, btrfs-progs usage of malloc() will result anonymous pages, thus
they can be swapped out.

Secondly, kernel doesn't like to swap out anonymous pages at all, thus
kernel won't try to aggressively swap out such pages.

Thirdly, for user anonymous memory, there is LRU-like algorithm to
determine which memory should go swapped out.
But considering how btrfs check uses pages, it would only make it harder
to be swapped out.

So it's not an easy way thing to aggressively swap out memory of btrfs
check to swap.
Although Michal mentioned some cgroup way to limit the memory usage so
it can be more aggressively swapped out, I'm still digging into it.

Thanks,
Qu

> 
> Thanks,
> Marc
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 479+ messages in thread

* 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
@ 2019-10-18  2:56       ` Marc MERLIN
  2019-10-19  3:07         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2019-10-18  2:56 UTC (permalink / raw)
  To: Linux BTRFS Mailinglist

This happened almost after a resume from suspend to disk.
First corruption and read only I got a very long time.

Could they be related?

[26062.126505] ------------[ cut here ]------------
[26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
[26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_hda
 _intel
[26062.126561]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
[26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
[26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
[26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921
[26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41
[26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246
[26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000
[26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558
[26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
[26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169
[26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
[26062.126625] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
[26062.126627] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
[26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[26062.126636] Call Trace:
[26062.126647]  __btrfs_run_delayed_refs+0x750/0xc36
[26062.126653]  ? __switch_to_asm+0x41/0x70
[26062.126655]  ? __switch_to_asm+0x35/0x70
[26062.126658]  ? __switch_to_asm+0x41/0x70
[26062.126662]  ? __switch_to+0x13d/0x3d5
[26062.126668]  btrfs_run_delayed_refs+0x5d/0x132
[26062.126672]  btrfs_commit_transaction+0x55/0x7c8
[26062.126676]  ? start_transaction+0x347/0x3cb
[26062.126679]  transaction_kthread+0xc9/0x135
[26062.126683]  ? btrfs_cleanup_transaction+0x403/0x403
[26062.126688]  kthread+0xeb/0xf0
[26062.126692]  ? kthread_create_worker_on_cpu+0x65/0x65
[26062.126695]  ret_from_fork+0x35/0x40
[26062.126698] ---[ end trace 4c1a6b3749a2f650 ]---
[26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2
[26062.126706] 	item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66
[26062.126708] 		extent refs 2 gen 2369265 flags 1
[26062.126709] 		ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1
[26062.126711] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126714] 	item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66
[26062.126715] 		extent refs 2 gen 2369025 flags 1
[26062.126716] 		ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1
[26062.126718] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126720] 	item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183
[26062.126722] 		extent refs 11 gen 1800715 flags 1
[26062.126722] 		ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1
[26062.126724] 		ref#1: shared data backref parent 508153839616 count 1
[26062.126726] 		ref#2: shared data backref parent 493541244928 count 1
[26062.126727] 		ref#3: shared data backref parent 492677332992 count 1
[26062.126728] 		ref#4: shared data backref parent 492328566784 count 1
[26062.126730] 		ref#5: shared data backref parent 492245188608 count 1
[26062.126731] 		ref#6: shared data backref parent 475885109248 count 1
[26062.126732] 		ref#7: shared data backref parent 471036608512 count 1
[26062.126733] 		ref#8: shared data backref parent 436114243584 count 1
[26062.126735] 		ref#9: shared data backref parent 886718464 count 1
[26062.126736] 		ref#10: shared data backref parent 90259456 count 1
[26062.126738] 	item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66
[26062.126740] 		extent refs 2 gen 2420471 flags 1
[26062.126741] 		ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1
[26062.126743] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126745] 	item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53
[26062.126746] 		extent refs 1 gen 2422941 flags 1
[26062.126747] 		ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1
[26062.126749] 	item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53
[26062.126750] 		extent refs 1 gen 1444854 flags 1
[26062.126751] 		ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1
[26062.126753] 	item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37
[26062.126754] 		extent refs 1 gen 1444854 flags 1
[26062.126755] 		ref#0: shared data backref parent 214892544 count 1
[26062.126757] 	item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37
[26062.126758] 		extent refs 1 gen 1444854 flags 1
[26062.126759] 		ref#0: shared data backref parent 102547456 count 1
[26062.126761] 	item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66
[26062.126762] 		extent refs 2 gen 2422644 flags 1
[26062.126763] 		ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1
[26062.126765] 		ref#1: shared data backref parent 470658482176 count 1
[26062.126768] 	item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66
[26062.126769] 		extent refs 2 gen 2369217 flags 1
[26062.126770] 		ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1
[26062.126772] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126774] 	item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66
[26062.126775] 		extent refs 2 gen 2369242 flags 1
[26062.126776] 		ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1
[26062.126778] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126780] 	item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53
[26062.126781] 		extent refs 1 gen 2369305 flags 1
[26062.126782] 		ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1
[26062.126784] 	item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66
[26062.126786] 		extent refs 2 gen 2369052 flags 1
[26062.126787] 		ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1
[26062.126788] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126790] 	item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53
[26062.126792] 		extent refs 1 gen 2369305 flags 1
[26062.126793] 		ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1
[26062.126795] 	item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66
[26062.126796] 		extent refs 2 gen 2368754 flags 1
[26062.126797] 		ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1
[26062.126799] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126801] 	item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183
[26062.126802] 		extent refs 11 gen 1800682 flags 1
[26062.126803] 		ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1
[26062.126805] 		ref#1: shared data backref parent 508153069568 count 1
[26062.126806] 		ref#2: shared data backref parent 508031270912 count 1
[26062.126808] 		ref#3: shared data backref parent 493564215296 count 1
[26062.126809] 		ref#4: shared data backref parent 493540098048 count 1
[26062.126811] 		ref#5: shared data backref parent 492676038656 count 1
[26062.126813] 		ref#6: shared data backref parent 475884142592 count 1
[26062.126814] 		ref#7: shared data backref parent 471032840192 count 1
[26062.126816] 		ref#8: shared data backref parent 436112146432 count 1
[26062.126817] 		ref#9: shared data backref parent 883752960 count 1
[26062.126818] 		ref#10: shared data backref parent 86654976 count 1
[26062.126820] 	item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50
[26062.126822] 		extent refs 2 gen 2369347 flags 1
[26062.126823] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126824] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126826] 	item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50
[26062.126827] 		extent refs 2 gen 2369347 flags 1
[26062.126828] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126830] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126832] 	item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50
[26062.126833] 		extent refs 2 gen 2369347 flags 1
[26062.126834] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126836] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126839] 	item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37
[26062.126840] 		extent refs 1 gen 2369352 flags 1
[26062.126841] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126843] 	item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37
[26062.126845] 		extent refs 1 gen 2369352 flags 1
[26062.126845] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126848] 	item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37
[26062.126849] 		extent refs 1 gen 2369352 flags 1
[26062.126850] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126852] 	item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50
[26062.126853] 		extent refs 2 gen 2369347 flags 1
[26062.126854] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126856] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126858] 	item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50
[26062.126859] 		extent refs 2 gen 2369347 flags 1
[26062.126860] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126862] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126864] 	item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50
[26062.126865] 		extent refs 2 gen 2369347 flags 1
[26062.126866] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126868] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126870] 	item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50
[26062.126871] 		extent refs 2 gen 2369347 flags 1
[26062.126873] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126874] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126876] 	item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50
[26062.126877] 		extent refs 2 gen 2369347 flags 1
[26062.126878] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126880] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126882] 	item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50
[26062.126883] 		extent refs 2 gen 2369347 flags 1
[26062.126884] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126885] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126888] 	item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50
[26062.126889] 		extent refs 2 gen 2369347 flags 1
[26062.126890] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126891] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126893] 	item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79
[26062.126894] 		extent refs 3 gen 2369347 flags 1
[26062.126895] 		ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1
[26062.126897] 		ref#1: shared data backref parent 492981370880 count 1
[26062.126898] 		ref#2: shared data backref parent 438017720320 count 1
[26062.126900] 	item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66
[26062.126902] 		extent refs 2 gen 2369347 flags 1
[26062.126903] 		ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1
[26062.126904] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126906] 	item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37
[26062.126907] 		extent refs 1 gen 2369352 flags 1
[26062.126908] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126910] 	item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79
[26062.126912] 		extent refs 6 gen 1800750 flags 1
[26062.126913] 		ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2
[26062.126915] 		ref#1: shared data backref parent 509782933504 count 2
[26062.126917] 		ref#2: shared data backref parent 508543795200 count 2
[26062.126919] 	item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66
[26062.126920] 		extent refs 2 gen 2369075 flags 1
[26062.126921] 		ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1
[26062.126923] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126925] 	item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79
[26062.126926] 		extent refs 3 gen 1800733 flags 1
[26062.126927] 		ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1
[26062.126929] 		ref#1: shared data backref parent 509783719936 count 1
[26062.126930] 		ref#2: shared data backref parent 509351231488 count 1
[26062.126932] 	item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66
[26062.126934] 		extent refs 2 gen 2420471 flags 1
[26062.126934] 		ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1
[26062.126936] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126938] 	item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66
[26062.126939] 		extent refs 2 gen 2420471 flags 1
[26062.126940] 		ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1
[26062.126943] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126945] 	item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79
[26062.126946] 		extent refs 3 gen 1800733 flags 1
[26062.126947] 		ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1
[26062.126949] 		ref#1: shared data backref parent 509783719936 count 1
[26062.126950] 		ref#2: shared data backref parent 509351231488 count 1
[26062.126953] 	item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66
[26062.126954] 		extent refs 2 gen 2420502 flags 1
[26062.126955] 		ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1
[26062.126957] 		ref#1: shared data backref parent 492274728960 count 1
[26062.126959] 	item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66
[26062.126961] 		extent refs 2 gen 2420478 flags 1
[26062.126962] 		ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1
[26062.126964] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126966] 	item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37
[26062.126967] 		extent refs 1 gen 1444854 flags 1
[26062.126968] 		ref#0: shared data backref parent 214908928 count 1
[26062.126970] 	item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53
[26062.126971] 		extent refs 1 gen 2376479 flags 1
[26062.126972] 		ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1
[26062.126975] 	item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53
[26062.126976] 		extent refs 1 gen 1444854 flags 1
[26062.126977] 		ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1
[26062.126979] 	item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37
[26062.126981] 		extent refs 1 gen 1444854 flags 1
[26062.126982] 		ref#0: shared data backref parent 93732864 count 1
[26062.126984] 	item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37
[26062.126985] 		extent refs 1 gen 1444854 flags 1
[26062.126986] 		ref#0: shared data backref parent 93732864 count 1
[26062.126988] 	item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37
[26062.126990] 		extent refs 1 gen 1444854 flags 1
[26062.126991] 		ref#0: shared data backref parent 93732864 count 1
[26062.126993] 	item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37
[26062.126994] 		extent refs 1 gen 1444854 flags 1
[26062.126995] 		ref#0: shared data backref parent 214908928 count 1
[26062.126997] 	item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92
[26062.126998] 		extent refs 4 gen 2419085 flags 1
[26062.126999] 		ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1
[26062.127001] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127002] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127004] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127006] 	item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79
[26062.127008] 		extent refs 3 gen 1800733 flags 1
[26062.127009] 		ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1
[26062.127011] 		ref#1: shared data backref parent 509783719936 count 1
[26062.127013] 		ref#2: shared data backref parent 509351231488 count 1
[26062.127015] 	item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66
[26062.127016] 		extent refs 2 gen 2368859 flags 1
[26062.127017] 		ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1
[26062.127019] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127021] 	item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53
[26062.127023] 		extent refs 1 gen 2376479 flags 1
[26062.127024] 		ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1
[26062.127026] 	item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183
[26062.127028] 		extent refs 11 gen 1800674 flags 1
[26062.127029] 		ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1
[26062.127031] 		ref#1: shared data backref parent 508153118720 count 1
[26062.127032] 		ref#2: shared data backref parent 508031533056 count 1
[26062.127034] 		ref#3: shared data backref parent 493564477440 count 1
[26062.127036] 		ref#4: shared data backref parent 493540179968 count 1
[26062.127038] 		ref#5: shared data backref parent 492675366912 count 1
[26062.127039] 		ref#6: shared data backref parent 475132116992 count 1
[26062.127041] 		ref#7: shared data backref parent 471032922112 count 1
[26062.127042] 		ref#8: shared data backref parent 436112244736 count 1
[26062.127044] 		ref#9: shared data backref parent 883998720 count 1
[26062.127046] 		ref#10: shared data backref parent 87015424 count 1
[26062.127048] 	item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37
[26062.127050] 		extent refs 1 gen 2408131 flags 1
[26062.127052] 		ref#0: shared data backref parent 509278633984 count 1
[26062.127055] 	item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183
[26062.127056] 		extent refs 11 gen 1800715 flags 1
[26062.127058] 		ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1
[26062.127059] 		ref#1: shared data backref parent 508152889344 count 1
[26062.127061] 		ref#2: shared data backref parent 508031139840 count 1
[26062.127062] 		ref#3: shared data backref parent 493563478016 count 1
[26062.127065] 		ref#4: shared data backref parent 493539983360 count 1
[26062.127066] 		ref#5: shared data backref parent 492674990080 count 1
[26062.127068] 		ref#6: shared data backref parent 475132166144 count 1
[26062.127070] 		ref#7: shared data backref parent 471032348672 count 1
[26062.127071] 		ref#8: shared data backref parent 436111769600 count 1
[26062.127073] 		ref#9: shared data backref parent 883359744 count 1
[26062.127074] 		ref#10: shared data backref parent 85082112 count 1
[26062.127077] 	item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92
[26062.127078] 		extent refs 4 gen 2419085 flags 1
[26062.127079] 		ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1
[26062.127085] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127088] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127089] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127093] 	item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66
[26062.127095] 		extent refs 2 gen 2421893 flags 1
[26062.127096] 		ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1
[26062.127099] 		ref#1: shared data backref parent 493703069696 count 1
[26062.127102] 	item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53
[26062.127104] 		extent refs 1 gen 1444854 flags 1
[26062.127105] 		ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1
[26062.127109] 	item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66
[26062.127111] 		extent refs 2 gen 2368700 flags 1
[26062.127112] 		ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1
[26062.127114] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127118] 	item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37
[26062.127120] 		extent refs 1 gen 1444854 flags 1
[26062.127121] 		ref#0: shared data backref parent 93732864 count 1
[26062.127125] 	item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37
[26062.127127] 		extent refs 1 gen 1444854 flags 1
[26062.127128] 		ref#0: shared data backref parent 214433792 count 1
[26062.127131] 	item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37
[26062.127132] 		extent refs 1 gen 1444854 flags 1
[26062.127133] 		ref#0: shared data backref parent 214908928 count 1
[26062.127136] 	item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79
[26062.127138] 		extent refs 3 gen 1800750 flags 1
[26062.127139] 		ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1
[26062.127141] 		ref#1: shared data backref parent 509783556096 count 1
[26062.127143] 		ref#2: shared data backref parent 508544974848 count 1
[26062.127147] 	item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66
[26062.127148] 		extent refs 2 gen 2368707 flags 1
[26062.127149] 		ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1
[26062.127151] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127153] 	item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183
[26062.127155] 		extent refs 11 gen 1800674 flags 1
[26062.127156] 		ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1
[26062.127157] 		ref#1: shared data backref parent 508152905728 count 1
[26062.127159] 		ref#2: shared data backref parent 508031057920 count 1
[26062.127160] 		ref#3: shared data backref parent 493563166720 count 1
[26062.127161] 		ref#4: shared data backref parent 493539934208 count 1
[26062.127163] 		ref#5: shared data backref parent 492674908160 count 1
[26062.127164] 		ref#6: shared data backref parent 475884208128 count 1
[26062.127165] 		ref#7: shared data backref parent 471032266752 count 1
[26062.127167] 		ref#8: shared data backref parent 436111720448 count 1
[26062.127168] 		ref#9: shared data backref parent 883179520 count 1
[26062.127170] 		ref#10: shared data backref parent 83017728 count 1
[26062.127172] 	item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37
[26062.127173] 		extent refs 1 gen 1444854 flags 1
[26062.127174] 		ref#0: shared data backref parent 214908928 count 1
[26062.127176] 	item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66
[26062.127177] 		extent refs 2 gen 2381729 flags 1
[26062.127178] 		ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1
[26062.127180] 		ref#1: shared data backref parent 437808545792 count 1
[26062.127182] 	item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66
[26062.127183] 		extent refs 2 gen 2369098 flags 1
[26062.127184] 		ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1
[26062.127186] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127188] 	item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66
[26062.127189] 		extent refs 2 gen 2368700 flags 1
[26062.127190] 		ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1
[26062.127191] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127193] 	item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37
[26062.127195] 		extent refs 1 gen 1444854 flags 1
[26062.127195] 		ref#0: shared data backref parent 102547456 count 1
[26062.127197] 	item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79
[26062.127198] 		extent refs 3 gen 1800750 flags 1
[26062.127199] 		ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1
[26062.127201] 		ref#1: shared data backref parent 509783408640 count 1
[26062.127202] 		ref#2: shared data backref parent 508543451136 count 1
[26062.127204] 	item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37
[26062.127205] 		extent refs 1 gen 1444854 flags 1
[26062.127206] 		ref#0: shared data backref parent 93732864 count 1
[26062.127208] 	item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37
[26062.127209] 		extent refs 1 gen 1444854 flags 1
[26062.127210] 		ref#0: shared data backref parent 93732864 count 1
[26062.127213] 	item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53
[26062.127214] 		extent refs 1 gen 1444854 flags 1
[26062.127215] 		ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1
[26062.127217] 	item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92
[26062.127219] 		extent refs 4 gen 2419085 flags 1
[26062.127220] 		ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1
[26062.127222] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127223] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127225] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127226] 	item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79
[26062.127228] 		extent refs 3 gen 1800750 flags 1
[26062.127229] 		ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1
[26062.127230] 		ref#1: shared data backref parent 509783408640 count 1
[26062.127232] 		ref#2: shared data backref parent 508543451136 count 1
[26062.127234] 	item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66
[26062.127235] 		extent refs 2 gen 2368731 flags 1
[26062.127236] 		ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1
[26062.127237] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127239] 	item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37
[26062.127241] 		extent refs 1 gen 1444854 flags 1
[26062.127241] 		ref#0: shared data backref parent 93732864 count 1
[26062.127243] 	item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37
[26062.127244] 		extent refs 1 gen 1444854 flags 1
[26062.127245] 		ref#0: shared data backref parent 93732864 count 1
[26062.127247] 	item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37
[26062.127249] 		extent refs 1 gen 1444854 flags 1
[26062.127249] 		ref#0: shared data backref parent 93732864 count 1
[26062.127252] 	item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37
[26062.127253] 		extent refs 1 gen 1444854 flags 1
[26062.127254] 		ref#0: shared data backref parent 93732864 count 1
[26062.127257] 	item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37
[26062.127258] 		extent refs 1 gen 1444854 flags 1
[26062.127259] 		ref#0: shared data backref parent 93732864 count 1
[26062.127262] 	item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53
[26062.127263] 		extent refs 1 gen 2381741 flags 1
[26062.127264] 		ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1
[26062.127266] 	item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66
[26062.127268] 		extent refs 2 gen 2368882 flags 1
[26062.127269] 		ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1
[26062.127270] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127272] 	item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66
[26062.127274] 		extent refs 2 gen 2368909 flags 1
[26062.127275] 		ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1
[26062.127277] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127279] 	item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66
[26062.127280] 		extent refs 2 gen 2369146 flags 1
[26062.127281] 		ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1
[26062.127283] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127285] 	item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37
[26062.127286] 		extent refs 1 gen 2369353 flags 1
[26062.127287] 		ref#0: shared data backref parent 510167498752 count 1
[26062.127289] 	item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79
[26062.127294] 		extent refs 3 gen 1800733 flags 1
[26062.127294] 		ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1
[26062.127296] 		ref#1: shared data backref parent 509783752704 count 1
[26062.127298] 		ref#2: shared data backref parent 509351313408 count 1
[26062.127299] 	item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183
[26062.127303] 		extent refs 22 gen 1800674 flags 1
[26062.127304] 		ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2
[26062.127306] 		ref#1: shared data backref parent 508152889344 count 2
[26062.127307] 		ref#2: shared data backref parent 508031139840 count 2
[26062.127308] 		ref#3: shared data backref parent 493563478016 count 2
[26062.127310] 		ref#4: shared data backref parent 493539983360 count 2
[26062.127311] 		ref#5: shared data backref parent 492674990080 count 2
[26062.127312] 		ref#6: shared data backref parent 475132166144 count 2
[26062.127314] 		ref#7: shared data backref parent 471032348672 count 2
[26062.127315] 		ref#8: shared data backref parent 436111769600 count 2
[26062.127317] 		ref#9: shared data backref parent 883359744 count 2
[26062.127318] 		ref#10: shared data backref parent 85082112 count 2
[26062.127320] 	item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183
[26062.127321] 		extent refs 33 gen 1800674 flags 1
[26062.127322] 		ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3
[26062.127324] 		ref#1: shared data backref parent 508152889344 count 3
[26062.127325] 		ref#2: shared data backref parent 508031139840 count 3
[26062.127327] 		ref#3: shared data backref parent 493563478016 count 3
[26062.127328] 		ref#4: shared data backref parent 493539983360 count 3
[26062.127329] 		ref#5: shared data backref parent 492674990080 count 3
[26062.127331] 		ref#6: shared data backref parent 475132166144 count 3
[26062.127333] 		ref#7: shared data backref parent 471032348672 count 3
[26062.127334] 		ref#8: shared data backref parent 436111769600 count 3
[26062.127335] 		ref#9: shared data backref parent 883359744 count 3
[26062.127337] 		ref#10: shared data backref parent 85082112 count 3
[26062.127338] 	item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92
[26062.127340] 		extent refs 4 gen 2419085 flags 1
[26062.127341] 		ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1
[26062.127342] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127344] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127345] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127347] 	item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66
[26062.127348] 		extent refs 2 gen 2368932 flags 1
[26062.127349] 		ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1
[26062.127351] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127353] 	item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37
[26062.127354] 		extent refs 1 gen 2369353 flags 1
[26062.127355] 		ref#0: shared data backref parent 510165155840 count 1
[26062.127356] 	item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66
[26062.127358] 		extent refs 2 gen 2369168 flags 1
[26062.127359] 		ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1
[26062.127360] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127362] 	item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53
[26062.127364] 		extent refs 1 gen 2369305 flags 1
[26062.127365] 		ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1
[26062.127367] 	item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92
[26062.127369] 		extent refs 4 gen 2419085 flags 1
[26062.127370] 		ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1
[26062.127372] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127374] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127375] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127377] 	item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66
[26062.127378] 		extent refs 2 gen 2368955 flags 1
[26062.127379] 		ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1
[26062.127382] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127384] 	item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37
[26062.127385] 		extent refs 1 gen 2369352 flags 1
[26062.127386] 		ref#0: shared data backref parent 510178295808 count 1
[26062.127388] 	item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37
[26062.127390] 		extent refs 1 gen 2369352 flags 1
[26062.127391] 		ref#0: shared data backref parent 510058790912 count 1
[26062.127393] 	item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37
[26062.127394] 		extent refs 1 gen 2369352 flags 1
[26062.127395] 		ref#0: shared data backref parent 510059184128 count 1
[26062.127398] 	item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53
[26062.127399] 		extent refs 1 gen 2378552 flags 1
[26062.127400] 		ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1
[26062.127403] 	item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53
[26062.127405] 		extent refs 1 gen 2378552 flags 1
[26062.127406] 		ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1
[26062.127408] 	item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53
[26062.127409] 		extent refs 1 gen 2381741 flags 1
[26062.127410] 		ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1
[26062.127413] 	item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66
[26062.127414] 		extent refs 2 gen 2368700 flags 1
[26062.127415] 		ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1
[26062.127416] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127419] 	item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66
[26062.127420] 		extent refs 2 gen 2368700 flags 1
[26062.127421] 		ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1
[26062.127423] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127425] 	item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66
[26062.127426] 		extent refs 2 gen 2368700 flags 1
[26062.127428] 		ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1
[26062.127430] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127432] 	item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183
[26062.127434] 		extent refs 11 gen 1800699 flags 1
[26062.127435] 		ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1
[26062.127437] 		ref#1: shared data backref parent 508152987648 count 1
[26062.127439] 		ref#2: shared data backref parent 508031172608 count 1
[26062.127440] 		ref#3: shared data backref parent 493563822080 count 1
[26062.127441] 		ref#4: shared data backref parent 493540130816 count 1
[26062.127443] 		ref#5: shared data backref parent 492675170304 count 1
[26062.127444] 		ref#6: shared data backref parent 475884584960 count 1
[26062.127446] 		ref#7: shared data backref parent 471032496128 count 1
[26062.127447] 		ref#8: shared data backref parent 436111884288 count 1
[26062.127448] 		ref#9: shared data backref parent 883703808 count 1
[26062.127450] 		ref#10: shared data backref parent 85671936 count 1
[26062.127452] 	item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37
[26062.127453] 		extent refs 1 gen 1444854 flags 1
[26062.127454] 		ref#0: shared data backref parent 214908928 count 1
[26062.127457] 	item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37
[26062.127458] 		extent refs 1 gen 2426255 flags 1
[26062.127459] 		ref#0: shared data backref parent 436271005696 count 1
[26062.127461] 	item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183
[26062.127463] 		extent refs 22 gen 1800674 flags 1
[26062.127464] 		ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2
[26062.127465] 		ref#1: shared data backref parent 508152987648 count 2
[26062.127467] 		ref#2: shared data backref parent 508031172608 count 2
[26062.127468] 		ref#3: shared data backref parent 493563822080 count 2
[26062.127470] 		ref#4: shared data backref parent 493540130816 count 2
[26062.127472] 		ref#5: shared data backref parent 492675170304 count 2
[26062.127473] 		ref#6: shared data backref parent 475884584960 count 2
[26062.127475] 		ref#7: shared data backref parent 471032496128 count 2
[26062.127476] 		ref#8: shared data backref parent 436111884288 count 2
[26062.127478] 		ref#9: shared data backref parent 883703808 count 2
[26062.127480] 		ref#10: shared data backref parent 85671936 count 2
[26062.127483] 	item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37
[26062.127486] 		extent refs 1 gen 1444854 flags 1
[26062.127488] 		ref#0: shared data backref parent 167198720 count 1
[26062.127491] 	item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66
[26062.127494] 		extent refs 2 gen 1400856 flags 1
[26062.127496] 		ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1
[26062.127498] 		ref#1: shared data backref parent 470954196992 count 1
[26062.127501] 	item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37
[26062.127503] 		extent refs 1 gen 1444854 flags 1
[26062.127504] 		ref#0: shared data backref parent 93732864 count 1
[26062.127507] 	item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53
[26062.127510] 		extent refs 1 gen 2381741 flags 1
[26062.127512] 		ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1
[26062.127518] 	item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66
[26062.127520] 		extent refs 2 gen 2368701 flags 1
[26062.127521] 		ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1
[26062.127523] 		ref#1: shared data backref parent 494107131904 count 1
[26062.127526] 	item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183
[26062.127527] 		extent refs 44 gen 1800674 flags 1
[26062.127528] 		ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4
[26062.127530] 		ref#1: shared data backref parent 508152987648 count 4
[26062.127532] 		ref#2: shared data backref parent 508031172608 count 4
[26062.127533] 		ref#3: shared data backref parent 493563822080 count 4
[26062.127535] 		ref#4: shared data backref parent 493540130816 count 4
[26062.127537] 		ref#5: shared data backref parent 492675170304 count 4
[26062.127539] 		ref#6: shared data backref parent 475884584960 count 4
[26062.127540] 		ref#7: shared data backref parent 471032496128 count 4
[26062.127542] 		ref#8: shared data backref parent 436111884288 count 4
[26062.127543] 		ref#9: shared data backref parent 883703808 count 4
[26062.127545] 		ref#10: shared data backref parent 85671936 count 4
[26062.127547] 	item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79
[26062.127549] 		extent refs 3 gen 1800750 flags 1
[26062.127552] 		ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1
[26062.127554] 		ref#1: shared data backref parent 509783130112 count 1
[26062.127557] 		ref#2: shared data backref parent 508544876544 count 1
[26062.127560] 	item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37
[26062.127561] 		extent refs 1 gen 1444854 flags 1
[26062.127562] 		ref#0: shared data backref parent 93732864 count 1
[26062.127565] 	item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37
[26062.127566] 		extent refs 1 gen 1444854 flags 1
[26062.127568] 		ref#0: shared data backref parent 214908928 count 1
[26062.127570] 	item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37
[26062.127572] 		extent refs 1 gen 1444854 flags 1
[26062.127573] 		ref#0: shared data backref parent 93732864 count 1
[26062.127577] 	item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24
[26062.127580] 		block group used 834060288 chunk_objectid 256 flags 1
[26062.127581] 	item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37
[26062.127583] 		extent refs 1 gen 1444854 flags 1
[26062.127584] 		ref#0: shared data backref parent 167198720 count 1
[26062.127586] 	item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37
[26062.127587] 		extent refs 1 gen 1444854 flags 1
[26062.127588] 		ref#0: shared data backref parent 167198720 count 1
[26062.127590] 	item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37
[26062.127591] 		extent refs 1 gen 1444854 flags 1
[26062.127592] 		ref#0: shared data backref parent 167198720 count 1
[26062.127594] 	item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37
[26062.127595] 		extent refs 1 gen 1444854 flags 1
[26062.127596] 		ref#0: shared data backref parent 167198720 count 1
[26062.127598] 	item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37
[26062.127599] 		extent refs 1 gen 1444854 flags 1
[26062.127600] 		ref#0: shared data backref parent 167198720 count 1
[26062.127602] 	item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37
[26062.127603] 		extent refs 1 gen 1444854 flags 1
[26062.127604] 		ref#0: shared data backref parent 167198720 count 1
[26062.127606] 	item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183
[26062.127607] 		extent refs 11 gen 2409230 flags 1
[26062.127608] 		ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1
[26062.127610] 		ref#1: shared data backref parent 508153675776 count 1
[26062.127611] 		ref#2: shared data backref parent 493566640128 count 1
[26062.127613] 		ref#3: shared data backref parent 493540999168 count 1
[26062.127614] 		ref#4: shared data backref parent 492676497408 count 1
[26062.127616] 		ref#5: shared data backref parent 475884634112 count 1
[26062.127617] 		ref#6: shared data backref parent 475691106304 count 1
[26062.127619] 		ref#7: shared data backref parent 471034494976 count 1
[26062.127620] 		ref#8: shared data backref parent 436113883136 count 1
[26062.127622] 		ref#9: shared data backref parent 886407168 count 1
[26062.127623] 		ref#10: shared data backref parent 89440256 count 1
[26062.127625] 	item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170
[26062.127626] 		extent refs 10 gen 2413323 flags 1
[26062.127629] 		ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1
[26062.127631] 		ref#1: shared data backref parent 508153675776 count 1
[26062.127633] 		ref#2: shared data backref parent 493566640128 count 1
[26062.127635] 		ref#3: shared data backref parent 493540999168 count 1
[26062.127636] 		ref#4: shared data backref parent 492676497408 count 1
[26062.127638] 		ref#5: shared data backref parent 475691106304 count 1
[26062.127640] 		ref#6: shared data backref parent 471034494976 count 1
[26062.127642] 		ref#7: shared data backref parent 436113883136 count 1
[26062.127643] 		ref#8: shared data backref parent 886407168 count 1
[26062.127645] 		ref#9: shared data backref parent 89440256 count 1
[26062.127647] 	item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37
[26062.127648] 		extent refs 1 gen 2407793 flags 1
[26062.127649] 		ref#0: shared data backref parent 475884634112 count 1
[26062.127653] 	item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183
[26062.127655] 		extent refs 11 gen 1800729 flags 1
[26062.127656] 		ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1
[26062.127658] 		ref#1: shared data backref parent 508153069568 count 1
[26062.127659] 		ref#2: shared data backref parent 508031270912 count 1
[26062.127661] 		ref#3: shared data backref parent 493564215296 count 1
[26062.127663] 		ref#4: shared data backref parent 493540098048 count 1
[26062.127664] 		ref#5: shared data backref parent 492676038656 count 1
[26062.127666] 		ref#6: shared data backref parent 475884142592 count 1
[26062.127668] 		ref#7: shared data backref parent 471032840192 count 1
[26062.127669] 		ref#8: shared data backref parent 436112146432 count 1
[26062.127671] 		ref#9: shared data backref parent 883752960 count 1
[26062.127674] 		ref#10: shared data backref parent 86654976 count 1
[26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456  owner 73020573 offset 17039360
[26062.127680] ------------[ cut here ]------------
[26062.127682] BTRFS: Transaction aborted (error -2)
[26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921
[26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_hda
 _intel
[26062.127733]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
[26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
[26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
[26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921
[26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df
[26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282
[26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007
[26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550
[26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
[26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169
[26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
[26062.127815] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
[26062.127817] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
[26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[26062.127823] Call Trace:
[26062.127832]  __btrfs_run_delayed_refs+0x750/0xc36
[26062.127837]  ? __switch_to_asm+0x41/0x70
[26062.127839]  ? __switch_to_asm+0x35/0x70
[26062.127841]  ? __switch_to_asm+0x41/0x70
[26062.127848]  ? __switch_to+0x13d/0x3d5
[26062.127852]  btrfs_run_delayed_refs+0x5d/0x132
[26062.127855]  btrfs_commit_transaction+0x55/0x7c8
[26062.127858]  ? start_transaction+0x347/0x3cb
[26062.127862]  transaction_kthread+0xc9/0x135
[26062.127865]  ? btrfs_cleanup_transaction+0x403/0x403
[26062.127869]  kthread+0xeb/0xf0
[26062.127872]  ? kthread_create_worker_on_cpu+0x65/0x65
[26062.127875]  ret_from_fork+0x35/0x40
[26062.127879] ---[ end trace 4c1a6b3749a2f651 ]---
[26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry
[26062.127917] BTRFS info (device dm-2): forced readonly
[26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entry

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
  2019-10-18  2:56       ` 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 Marc MERLIN
@ 2019-10-19  3:07         ` Marc MERLIN
  2019-10-23  0:55           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2019-10-19  3:07 UTC (permalink / raw)
  To: Linux BTRFS Mailinglist, Qu Wenruo, Su Yue; +Cc: Su Yue

Ok, so before blowing the filesystem away after it was apparently badly
damaged by a suspend to disk, I tried check --repair and I hit an
infinite loop.

Let me know if you'd like anything off the FS before I delete it.

Thanks,
Marc

enabling repair mode
repair mode will force to clear out log tree, are you sure? [y/N]: y
Checking filesystem on /dev/mapper/pool1
UUID: fda628bc-1ca4-49c5-91c2-4260fe967a23
checking extents
Backref 415334400 parent 36028797198598144 not referenced back 0x5648ef1870e0
Backref 415334400 parent 179634176 root 179634176 not found in extent tree
Incorrect global backref count on 415334400 found 2 wanted 1
backpointer mismatch on [415334400 16384]
repair deleting extent record: key 415334400 169 0
adding new tree backref on start 415334400 len 16384 parent 179634176 root 179634176
Repaired extent references for 415334400
ref mismatch on [101995261952 4096] extent item 36028797018963969, found 1
repair deleting extent record: key 101995261952 168 4096
adding new data backref on 101995261952 root 456 owner 74455677 offset 64892928 found 1
Repaired extent references for 101995261952
Incorrect local backref count on 458640384000 root 456 owner 81409181 offset 17039360 found 0 wanted 1 back 0x5648eefd3d10
Backref disk bytenr does not match extent record, bytenr=458640384000, ref bytenr=0
Backref 458640384000 root 456 owner 73020573 offset 17039360 num_refs 0 not found in extent tree
Incorrect local backref count on 458640384000 root 456 owner 73020573 offset 17039360 found 1 wanted 0 back 0x5648b32a9600
backpointer mismatch on [458640384000 86016]
repair deleting extent record: key 458640384000 168 86016
adding new data backref on 458640384000 parent 438017720320 owner 0 offset 0 found 1
adding new data backref on 458640384000 root 456 owner 73020573 offset 17039360 found 1
Repaired extent references for 458640384000
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
Deleting bad dir index [10138517,96,436945] root 456
Deleting bad dir index [10138518,96,646273] root 456
Deleting bad dir index [10138517,96,437016] root 456
Deleting bad dir index [10138518,96,808999] root 456
Deleting bad dir index [10215134,96,149427] root 456
Deleting bad dir index [10240541,96,268037] root 456
Deleting bad dir index [10138517,96,540247] root 456
Deleting bad dir index [10138518,96,825234] root 456
Deleting bad dir index [10138517,96,736673] root 456
Deleting bad dir index [10138518,96,1118221] root 456
Deleting bad dir index [10240541,96,439703] root 456
Deleting bad dir index [10138517,96,752282] root 456
root 456 inode 75431563 errors 100, file extent discount
Found file extent holes:
	start: 4096, len: 4096
root 456 inode 75431568 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 1638400
root 456 inode 75431583 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 147456
root 456 inode 75431585 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 208896
root 456 inode 75431591 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 2523136
root 456 inode 75431730 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 208896
root 456 inode 75431744 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 2084864
root 456 inode 75431751 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 172032
root 456 inode 75431756 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
root 456 inode 75431760 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 12288
root 456 inode 75431765 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 32768
root 456 inode 75431773 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 90112
Fixed discount file extents for inode: 75432421 in root: 456
Fixed discount file extents for inode: 75432429 in root: 456
root 456 inode 75432429 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432430 in root: 456
Fixed discount file extents for inode: 75432432 in root: 456
Fixed discount file extents for inode: 75432433 in root: 456
root 456 inode 75432433 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
root 456 inode 75432434 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 724992
Fixed discount file extents for inode: 75432436 in root: 456
Fixed discount file extents for inode: 75432437 in root: 456
root 456 inode 75432437 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432438 in root: 456
Fixed discount file extents for inode: 75432441 in root: 456
root 456 inode 75432441 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432444 in root: 456
root 456 inode 75432444 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432445 in root: 456
Fixed discount file extents for inode: 75432447 in root: 456
Fixed discount file extents for inode: 75432448 in root: 456
Fixed discount file extents for inode: 75432454 in root: 456
Fixed discount file extents for inode: 75432457 in root: 456
root 456 inode 75432457 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
root 456 inode 75432471 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 20480
Fixed discount file extents for inode: 75432473 in root: 456
root 456 inode 75432473 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 20480
Fixed discount file extents for inode: 75432482 in root: 456
Fixed discount file extents for inode: 75432483 in root: 456
Fixed discount file extents for inode: 75432484 in root: 456
Fixed discount file extents for inode: 75432487 in root: 456
Fixed discount file extents for inode: 75432489 in root: 456
Fixed discount file extents for inode: 75432494 in root: 456
Fixed discount file extents for inode: 75432611 in root: 456
Fixed discount file extents for inode: 75432622 in root: 456
Fixed discount file extents for inode: 75432626 in root: 456
Fixed discount file extents for inode: 75432628 in root: 456
Fixed discount file extents for inode: 75432636 in root: 456
Fixed discount file extents for inode: 75432637 in root: 456
root 456 inode 75432637 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432643 in root: 456
root 456 inode 75432643 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 135168
Fixed discount file extents for inode: 75432650 in root: 456
root 456 inode 75432650 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432656 in root: 456
root 456 inode 75432656 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432657 in root: 456
root 456 inode 75432657 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432659 in root: 456
Fixed discount file extents for inode: 75432662 in root: 456
root 456 inode 75432662 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432663 in root: 456
root 456 inode 75432663 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432664 in root: 456
root 456 inode 75432664 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432674 in root: 456
root 456 inode 75432674 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432692 in root: 456
root 456 inode 75432692 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432708 in root: 456
Fixed discount file extents for inode: 75432709 in root: 456
root 456 inode 75432709 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432712 in root: 456
Fixed discount file extents for inode: 75432717 in root: 456
Fixed discount file extents for inode: 75432720 in root: 456
Fixed discount file extents for inode: 75432725 in root: 456
Fixed discount file extents for inode: 75432731 in root: 456
Fixed discount file extents for inode: 75432732 in root: 456
Fixed discount file extents for inode: 75432749 in root: 456
Fixed discount file extents for inode: 75432762 in root: 456
Fixed discount file extents for inode: 75432765 in root: 456
Fixed discount file extents for inode: 75432783 in root: 456
Fixed discount file extents for inode: 75432784 in root: 456
Fixed discount file extents for inode: 75432785 in root: 456
root 456 inode 75432785 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432786 in root: 456
root 456 inode 75432786 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432796 in root: 456
root 456 inode 75432796 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432799 in root: 456
root 456 inode 75432799 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432801 in root: 456
root 456 inode 75432801 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432807 in root: 456
Fixed discount file extents for inode: 75432817 in root: 456
Fixed discount file extents for inode: 75432829 in root: 456
Fixed discount file extents for inode: 75432860 in root: 456
root 456 inode 75432860 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432862 in root: 456
Fixed discount file extents for inode: 75432863 in root: 456
Fixed discount file extents for inode: 75432869 in root: 456
Fixed discount file extents for inode: 75432870 in root: 456
Fixed discount file extents for inode: 75432871 in root: 456
Fixed discount file extents for inode: 75432872 in root: 456
Fixed discount file extents for inode: 75432875 in root: 456
Fixed discount file extents for inode: 75432877 in root: 456
Fixed discount file extents for inode: 75432882 in root: 456
Fixed discount file extents for inode: 75432883 in root: 456
Fixed discount file extents for inode: 75432893 in root: 456
Fixed discount file extents for inode: 75432894 in root: 456
Fixed discount file extents for inode: 75432897 in root: 456
Fixed discount file extents for inode: 75432899 in root: 456
Fixed discount file extents for inode: 75432900 in root: 456
Fixed discount file extents for inode: 75432905 in root: 456
Fixed discount file extents for inode: 75432906 in root: 456
Fixed discount file extents for inode: 75432916 in root: 456
Fixed discount file extents for inode: 75432917 in root: 456
Fixed discount file extents for inode: 75432919 in root: 456
Fixed discount file extents for inode: 75432920 in root: 456
Fixed discount file extents for inode: 75432923 in root: 456
Fixed discount file extents for inode: 75432942 in root: 456
Fixed discount file extents for inode: 75432944 in root: 456
Fixed discount file extents for inode: 75432948 in root: 456
root 456 inode 75432948 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432949 in root: 456
root 456 inode 75432949 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
and it loops forever on 456

On Thu, Oct 17, 2019 at 07:56:04PM -0700, Marc MERLIN wrote:
> This happened almost after a resume from suspend to disk.
> First corruption and read only I got a very long time.
> 
> Could they be related?
> 
> [26062.126505] ------------[ cut here ]------------
> [26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
> [26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_h
 da_intel
> [26062.126561]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
> [26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
> [26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
> [26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921
> [26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41
> [26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246
> [26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000
> [26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558
> [26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
> [26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169
> [26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
> [26062.126625] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
> [26062.126627] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
> [26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [26062.126636] Call Trace:
> [26062.126647]  __btrfs_run_delayed_refs+0x750/0xc36
> [26062.126653]  ? __switch_to_asm+0x41/0x70
> [26062.126655]  ? __switch_to_asm+0x35/0x70
> [26062.126658]  ? __switch_to_asm+0x41/0x70
> [26062.126662]  ? __switch_to+0x13d/0x3d5
> [26062.126668]  btrfs_run_delayed_refs+0x5d/0x132
> [26062.126672]  btrfs_commit_transaction+0x55/0x7c8
> [26062.126676]  ? start_transaction+0x347/0x3cb
> [26062.126679]  transaction_kthread+0xc9/0x135
> [26062.126683]  ? btrfs_cleanup_transaction+0x403/0x403
> [26062.126688]  kthread+0xeb/0xf0
> [26062.126692]  ? kthread_create_worker_on_cpu+0x65/0x65
> [26062.126695]  ret_from_fork+0x35/0x40
> [26062.126698] ---[ end trace 4c1a6b3749a2f650 ]---
> [26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2
> [26062.126706] 	item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66
> [26062.126708] 		extent refs 2 gen 2369265 flags 1
> [26062.126709] 		ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1
> [26062.126711] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.126714] 	item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66
> [26062.126715] 		extent refs 2 gen 2369025 flags 1
> [26062.126716] 		ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1
> [26062.126718] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.126720] 	item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183
> [26062.126722] 		extent refs 11 gen 1800715 flags 1
> [26062.126722] 		ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1
> [26062.126724] 		ref#1: shared data backref parent 508153839616 count 1
> [26062.126726] 		ref#2: shared data backref parent 493541244928 count 1
> [26062.126727] 		ref#3: shared data backref parent 492677332992 count 1
> [26062.126728] 		ref#4: shared data backref parent 492328566784 count 1
> [26062.126730] 		ref#5: shared data backref parent 492245188608 count 1
> [26062.126731] 		ref#6: shared data backref parent 475885109248 count 1
> [26062.126732] 		ref#7: shared data backref parent 471036608512 count 1
> [26062.126733] 		ref#8: shared data backref parent 436114243584 count 1
> [26062.126735] 		ref#9: shared data backref parent 886718464 count 1
> [26062.126736] 		ref#10: shared data backref parent 90259456 count 1
> [26062.126738] 	item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66
> [26062.126740] 		extent refs 2 gen 2420471 flags 1
> [26062.126741] 		ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1
> [26062.126743] 		ref#1: shared data backref parent 41020112896 count 1
> [26062.126745] 	item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53
> [26062.126746] 		extent refs 1 gen 2422941 flags 1
> [26062.126747] 		ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1
> [26062.126749] 	item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53
> [26062.126750] 		extent refs 1 gen 1444854 flags 1
> [26062.126751] 		ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1
> [26062.126753] 	item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37
> [26062.126754] 		extent refs 1 gen 1444854 flags 1
> [26062.126755] 		ref#0: shared data backref parent 214892544 count 1
> [26062.126757] 	item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37
> [26062.126758] 		extent refs 1 gen 1444854 flags 1
> [26062.126759] 		ref#0: shared data backref parent 102547456 count 1
> [26062.126761] 	item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66
> [26062.126762] 		extent refs 2 gen 2422644 flags 1
> [26062.126763] 		ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1
> [26062.126765] 		ref#1: shared data backref parent 470658482176 count 1
> [26062.126768] 	item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66
> [26062.126769] 		extent refs 2 gen 2369217 flags 1
> [26062.126770] 		ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1
> [26062.126772] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.126774] 	item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66
> [26062.126775] 		extent refs 2 gen 2369242 flags 1
> [26062.126776] 		ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1
> [26062.126778] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.126780] 	item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53
> [26062.126781] 		extent refs 1 gen 2369305 flags 1
> [26062.126782] 		ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1
> [26062.126784] 	item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66
> [26062.126786] 		extent refs 2 gen 2369052 flags 1
> [26062.126787] 		ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1
> [26062.126788] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.126790] 	item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53
> [26062.126792] 		extent refs 1 gen 2369305 flags 1
> [26062.126793] 		ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1
> [26062.126795] 	item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66
> [26062.126796] 		extent refs 2 gen 2368754 flags 1
> [26062.126797] 		ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1
> [26062.126799] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.126801] 	item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183
> [26062.126802] 		extent refs 11 gen 1800682 flags 1
> [26062.126803] 		ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1
> [26062.126805] 		ref#1: shared data backref parent 508153069568 count 1
> [26062.126806] 		ref#2: shared data backref parent 508031270912 count 1
> [26062.126808] 		ref#3: shared data backref parent 493564215296 count 1
> [26062.126809] 		ref#4: shared data backref parent 493540098048 count 1
> [26062.126811] 		ref#5: shared data backref parent 492676038656 count 1
> [26062.126813] 		ref#6: shared data backref parent 475884142592 count 1
> [26062.126814] 		ref#7: shared data backref parent 471032840192 count 1
> [26062.126816] 		ref#8: shared data backref parent 436112146432 count 1
> [26062.126817] 		ref#9: shared data backref parent 883752960 count 1
> [26062.126818] 		ref#10: shared data backref parent 86654976 count 1
> [26062.126820] 	item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50
> [26062.126822] 		extent refs 2 gen 2369347 flags 1
> [26062.126823] 		ref#0: shared data backref parent 492981370880 count 1
> [26062.126824] 		ref#1: shared data backref parent 438017720320 count 1
> [26062.126826] 	item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50
> [26062.126827] 		extent refs 2 gen 2369347 flags 1
> [26062.126828] 		ref#0: shared data backref parent 492981370880 count 1
> [26062.126830] 		ref#1: shared data backref parent 438017720320 count 1
> [26062.126832] 	item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50
> [26062.126833] 		extent refs 2 gen 2369347 flags 1
> [26062.126834] 		ref#0: shared data backref parent 492981370880 count 1
> [26062.126836] 		ref#1: shared data backref parent 438017720320 count 1
> [26062.126839] 	item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37
> [26062.126840] 		extent refs 1 gen 2369352 flags 1
> [26062.126841] 		ref#0: shared data backref parent 510058790912 count 1
> [26062.126843] 	item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37
> [26062.126845] 		extent refs 1 gen 2369352 flags 1
> [26062.126845] 		ref#0: shared data backref parent 510058790912 count 1
> [26062.126848] 	item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37
> [26062.126849] 		extent refs 1 gen 2369352 flags 1
> [26062.126850] 		ref#0: shared data backref parent 510058790912 count 1
> [26062.126852] 	item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50
> [26062.126853] 		extent refs 2 gen 2369347 flags 1
> [26062.126854] 		ref#0: shared data backref parent 492981370880 count 1
> [26062.126856] 		ref#1: shared data backref parent 438017720320 count 1
> [26062.126858] 	item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50
> [26062.126859] 		extent refs 2 gen 2369347 flags 1
> [26062.126860] 		ref#0: shared data backref parent 492981370880 count 1
> [26062.126862] 		ref#1: shared data backref parent 438017720320 count 1
> [26062.126864] 	item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50
> [26062.126865] 		extent refs 2 gen 2369347 flags 1
> [26062.126866] 		ref#0: shared data backref parent 492981370880 count 1
> [26062.126868] 		ref#1: shared data backref parent 438017720320 count 1
> [26062.126870] 	item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50
> [26062.126871] 		extent refs 2 gen 2369347 flags 1
> [26062.126873] 		ref#0: shared data backref parent 492981370880 count 1
> [26062.126874] 		ref#1: shared data backref parent 438017720320 count 1
> [26062.126876] 	item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50
> [26062.126877] 		extent refs 2 gen 2369347 flags 1
> [26062.126878] 		ref#0: shared data backref parent 492981370880 count 1
> [26062.126880] 		ref#1: shared data backref parent 438017720320 count 1
> [26062.126882] 	item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50
> [26062.126883] 		extent refs 2 gen 2369347 flags 1
> [26062.126884] 		ref#0: shared data backref parent 492981370880 count 1
> [26062.126885] 		ref#1: shared data backref parent 438017720320 count 1
> [26062.126888] 	item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50
> [26062.126889] 		extent refs 2 gen 2369347 flags 1
> [26062.126890] 		ref#0: shared data backref parent 492981370880 count 1
> [26062.126891] 		ref#1: shared data backref parent 438017720320 count 1
> [26062.126893] 	item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79
> [26062.126894] 		extent refs 3 gen 2369347 flags 1
> [26062.126895] 		ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1
> [26062.126897] 		ref#1: shared data backref parent 492981370880 count 1
> [26062.126898] 		ref#2: shared data backref parent 438017720320 count 1
> [26062.126900] 	item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66
> [26062.126902] 		extent refs 2 gen 2369347 flags 1
> [26062.126903] 		ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1
> [26062.126904] 		ref#1: shared data backref parent 438017720320 count 1
> [26062.126906] 	item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37
> [26062.126907] 		extent refs 1 gen 2369352 flags 1
> [26062.126908] 		ref#0: shared data backref parent 510058790912 count 1
> [26062.126910] 	item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79
> [26062.126912] 		extent refs 6 gen 1800750 flags 1
> [26062.126913] 		ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2
> [26062.126915] 		ref#1: shared data backref parent 509782933504 count 2
> [26062.126917] 		ref#2: shared data backref parent 508543795200 count 2
> [26062.126919] 	item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66
> [26062.126920] 		extent refs 2 gen 2369075 flags 1
> [26062.126921] 		ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1
> [26062.126923] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.126925] 	item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79
> [26062.126926] 		extent refs 3 gen 1800733 flags 1
> [26062.126927] 		ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1
> [26062.126929] 		ref#1: shared data backref parent 509783719936 count 1
> [26062.126930] 		ref#2: shared data backref parent 509351231488 count 1
> [26062.126932] 	item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66
> [26062.126934] 		extent refs 2 gen 2420471 flags 1
> [26062.126934] 		ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1
> [26062.126936] 		ref#1: shared data backref parent 41020112896 count 1
> [26062.126938] 	item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66
> [26062.126939] 		extent refs 2 gen 2420471 flags 1
> [26062.126940] 		ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1
> [26062.126943] 		ref#1: shared data backref parent 41020112896 count 1
> [26062.126945] 	item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79
> [26062.126946] 		extent refs 3 gen 1800733 flags 1
> [26062.126947] 		ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1
> [26062.126949] 		ref#1: shared data backref parent 509783719936 count 1
> [26062.126950] 		ref#2: shared data backref parent 509351231488 count 1
> [26062.126953] 	item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66
> [26062.126954] 		extent refs 2 gen 2420502 flags 1
> [26062.126955] 		ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1
> [26062.126957] 		ref#1: shared data backref parent 492274728960 count 1
> [26062.126959] 	item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66
> [26062.126961] 		extent refs 2 gen 2420478 flags 1
> [26062.126962] 		ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1
> [26062.126964] 		ref#1: shared data backref parent 41020112896 count 1
> [26062.126966] 	item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37
> [26062.126967] 		extent refs 1 gen 1444854 flags 1
> [26062.126968] 		ref#0: shared data backref parent 214908928 count 1
> [26062.126970] 	item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53
> [26062.126971] 		extent refs 1 gen 2376479 flags 1
> [26062.126972] 		ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1
> [26062.126975] 	item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53
> [26062.126976] 		extent refs 1 gen 1444854 flags 1
> [26062.126977] 		ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1
> [26062.126979] 	item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37
> [26062.126981] 		extent refs 1 gen 1444854 flags 1
> [26062.126982] 		ref#0: shared data backref parent 93732864 count 1
> [26062.126984] 	item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37
> [26062.126985] 		extent refs 1 gen 1444854 flags 1
> [26062.126986] 		ref#0: shared data backref parent 93732864 count 1
> [26062.126988] 	item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37
> [26062.126990] 		extent refs 1 gen 1444854 flags 1
> [26062.126991] 		ref#0: shared data backref parent 93732864 count 1
> [26062.126993] 	item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37
> [26062.126994] 		extent refs 1 gen 1444854 flags 1
> [26062.126995] 		ref#0: shared data backref parent 214908928 count 1
> [26062.126997] 	item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92
> [26062.126998] 		extent refs 4 gen 2419085 flags 1
> [26062.126999] 		ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1
> [26062.127001] 		ref#1: shared data backref parent 494075035648 count 1
> [26062.127002] 		ref#2: shared data backref parent 470850240512 count 1
> [26062.127004] 		ref#3: shared data backref parent 437034745856 count 1
> [26062.127006] 	item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79
> [26062.127008] 		extent refs 3 gen 1800733 flags 1
> [26062.127009] 		ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1
> [26062.127011] 		ref#1: shared data backref parent 509783719936 count 1
> [26062.127013] 		ref#2: shared data backref parent 509351231488 count 1
> [26062.127015] 	item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66
> [26062.127016] 		extent refs 2 gen 2368859 flags 1
> [26062.127017] 		ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1
> [26062.127019] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.127021] 	item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53
> [26062.127023] 		extent refs 1 gen 2376479 flags 1
> [26062.127024] 		ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1
> [26062.127026] 	item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183
> [26062.127028] 		extent refs 11 gen 1800674 flags 1
> [26062.127029] 		ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1
> [26062.127031] 		ref#1: shared data backref parent 508153118720 count 1
> [26062.127032] 		ref#2: shared data backref parent 508031533056 count 1
> [26062.127034] 		ref#3: shared data backref parent 493564477440 count 1
> [26062.127036] 		ref#4: shared data backref parent 493540179968 count 1
> [26062.127038] 		ref#5: shared data backref parent 492675366912 count 1
> [26062.127039] 		ref#6: shared data backref parent 475132116992 count 1
> [26062.127041] 		ref#7: shared data backref parent 471032922112 count 1
> [26062.127042] 		ref#8: shared data backref parent 436112244736 count 1
> [26062.127044] 		ref#9: shared data backref parent 883998720 count 1
> [26062.127046] 		ref#10: shared data backref parent 87015424 count 1
> [26062.127048] 	item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37
> [26062.127050] 		extent refs 1 gen 2408131 flags 1
> [26062.127052] 		ref#0: shared data backref parent 509278633984 count 1
> [26062.127055] 	item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183
> [26062.127056] 		extent refs 11 gen 1800715 flags 1
> [26062.127058] 		ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1
> [26062.127059] 		ref#1: shared data backref parent 508152889344 count 1
> [26062.127061] 		ref#2: shared data backref parent 508031139840 count 1
> [26062.127062] 		ref#3: shared data backref parent 493563478016 count 1
> [26062.127065] 		ref#4: shared data backref parent 493539983360 count 1
> [26062.127066] 		ref#5: shared data backref parent 492674990080 count 1
> [26062.127068] 		ref#6: shared data backref parent 475132166144 count 1
> [26062.127070] 		ref#7: shared data backref parent 471032348672 count 1
> [26062.127071] 		ref#8: shared data backref parent 436111769600 count 1
> [26062.127073] 		ref#9: shared data backref parent 883359744 count 1
> [26062.127074] 		ref#10: shared data backref parent 85082112 count 1
> [26062.127077] 	item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92
> [26062.127078] 		extent refs 4 gen 2419085 flags 1
> [26062.127079] 		ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1
> [26062.127085] 		ref#1: shared data backref parent 494075035648 count 1
> [26062.127088] 		ref#2: shared data backref parent 470850240512 count 1
> [26062.127089] 		ref#3: shared data backref parent 437034745856 count 1
> [26062.127093] 	item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66
> [26062.127095] 		extent refs 2 gen 2421893 flags 1
> [26062.127096] 		ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1
> [26062.127099] 		ref#1: shared data backref parent 493703069696 count 1
> [26062.127102] 	item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53
> [26062.127104] 		extent refs 1 gen 1444854 flags 1
> [26062.127105] 		ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1
> [26062.127109] 	item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66
> [26062.127111] 		extent refs 2 gen 2368700 flags 1
> [26062.127112] 		ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1
> [26062.127114] 		ref#1: shared data backref parent 494105804800 count 1
> [26062.127118] 	item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37
> [26062.127120] 		extent refs 1 gen 1444854 flags 1
> [26062.127121] 		ref#0: shared data backref parent 93732864 count 1
> [26062.127125] 	item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37
> [26062.127127] 		extent refs 1 gen 1444854 flags 1
> [26062.127128] 		ref#0: shared data backref parent 214433792 count 1
> [26062.127131] 	item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37
> [26062.127132] 		extent refs 1 gen 1444854 flags 1
> [26062.127133] 		ref#0: shared data backref parent 214908928 count 1
> [26062.127136] 	item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79
> [26062.127138] 		extent refs 3 gen 1800750 flags 1
> [26062.127139] 		ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1
> [26062.127141] 		ref#1: shared data backref parent 509783556096 count 1
> [26062.127143] 		ref#2: shared data backref parent 508544974848 count 1
> [26062.127147] 	item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66
> [26062.127148] 		extent refs 2 gen 2368707 flags 1
> [26062.127149] 		ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1
> [26062.127151] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.127153] 	item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183
> [26062.127155] 		extent refs 11 gen 1800674 flags 1
> [26062.127156] 		ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1
> [26062.127157] 		ref#1: shared data backref parent 508152905728 count 1
> [26062.127159] 		ref#2: shared data backref parent 508031057920 count 1
> [26062.127160] 		ref#3: shared data backref parent 493563166720 count 1
> [26062.127161] 		ref#4: shared data backref parent 493539934208 count 1
> [26062.127163] 		ref#5: shared data backref parent 492674908160 count 1
> [26062.127164] 		ref#6: shared data backref parent 475884208128 count 1
> [26062.127165] 		ref#7: shared data backref parent 471032266752 count 1
> [26062.127167] 		ref#8: shared data backref parent 436111720448 count 1
> [26062.127168] 		ref#9: shared data backref parent 883179520 count 1
> [26062.127170] 		ref#10: shared data backref parent 83017728 count 1
> [26062.127172] 	item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37
> [26062.127173] 		extent refs 1 gen 1444854 flags 1
> [26062.127174] 		ref#0: shared data backref parent 214908928 count 1
> [26062.127176] 	item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66
> [26062.127177] 		extent refs 2 gen 2381729 flags 1
> [26062.127178] 		ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1
> [26062.127180] 		ref#1: shared data backref parent 437808545792 count 1
> [26062.127182] 	item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66
> [26062.127183] 		extent refs 2 gen 2369098 flags 1
> [26062.127184] 		ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1
> [26062.127186] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.127188] 	item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66
> [26062.127189] 		extent refs 2 gen 2368700 flags 1
> [26062.127190] 		ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1
> [26062.127191] 		ref#1: shared data backref parent 494105804800 count 1
> [26062.127193] 	item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37
> [26062.127195] 		extent refs 1 gen 1444854 flags 1
> [26062.127195] 		ref#0: shared data backref parent 102547456 count 1
> [26062.127197] 	item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79
> [26062.127198] 		extent refs 3 gen 1800750 flags 1
> [26062.127199] 		ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1
> [26062.127201] 		ref#1: shared data backref parent 509783408640 count 1
> [26062.127202] 		ref#2: shared data backref parent 508543451136 count 1
> [26062.127204] 	item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37
> [26062.127205] 		extent refs 1 gen 1444854 flags 1
> [26062.127206] 		ref#0: shared data backref parent 93732864 count 1
> [26062.127208] 	item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37
> [26062.127209] 		extent refs 1 gen 1444854 flags 1
> [26062.127210] 		ref#0: shared data backref parent 93732864 count 1
> [26062.127213] 	item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53
> [26062.127214] 		extent refs 1 gen 1444854 flags 1
> [26062.127215] 		ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1
> [26062.127217] 	item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92
> [26062.127219] 		extent refs 4 gen 2419085 flags 1
> [26062.127220] 		ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1
> [26062.127222] 		ref#1: shared data backref parent 494075035648 count 1
> [26062.127223] 		ref#2: shared data backref parent 470850240512 count 1
> [26062.127225] 		ref#3: shared data backref parent 437034745856 count 1
> [26062.127226] 	item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79
> [26062.127228] 		extent refs 3 gen 1800750 flags 1
> [26062.127229] 		ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1
> [26062.127230] 		ref#1: shared data backref parent 509783408640 count 1
> [26062.127232] 		ref#2: shared data backref parent 508543451136 count 1
> [26062.127234] 	item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66
> [26062.127235] 		extent refs 2 gen 2368731 flags 1
> [26062.127236] 		ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1
> [26062.127237] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.127239] 	item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37
> [26062.127241] 		extent refs 1 gen 1444854 flags 1
> [26062.127241] 		ref#0: shared data backref parent 93732864 count 1
> [26062.127243] 	item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37
> [26062.127244] 		extent refs 1 gen 1444854 flags 1
> [26062.127245] 		ref#0: shared data backref parent 93732864 count 1
> [26062.127247] 	item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37
> [26062.127249] 		extent refs 1 gen 1444854 flags 1
> [26062.127249] 		ref#0: shared data backref parent 93732864 count 1
> [26062.127252] 	item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37
> [26062.127253] 		extent refs 1 gen 1444854 flags 1
> [26062.127254] 		ref#0: shared data backref parent 93732864 count 1
> [26062.127257] 	item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37
> [26062.127258] 		extent refs 1 gen 1444854 flags 1
> [26062.127259] 		ref#0: shared data backref parent 93732864 count 1
> [26062.127262] 	item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53
> [26062.127263] 		extent refs 1 gen 2381741 flags 1
> [26062.127264] 		ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1
> [26062.127266] 	item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66
> [26062.127268] 		extent refs 2 gen 2368882 flags 1
> [26062.127269] 		ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1
> [26062.127270] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.127272] 	item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66
> [26062.127274] 		extent refs 2 gen 2368909 flags 1
> [26062.127275] 		ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1
> [26062.127277] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.127279] 	item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66
> [26062.127280] 		extent refs 2 gen 2369146 flags 1
> [26062.127281] 		ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1
> [26062.127283] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.127285] 	item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37
> [26062.127286] 		extent refs 1 gen 2369353 flags 1
> [26062.127287] 		ref#0: shared data backref parent 510167498752 count 1
> [26062.127289] 	item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79
> [26062.127294] 		extent refs 3 gen 1800733 flags 1
> [26062.127294] 		ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1
> [26062.127296] 		ref#1: shared data backref parent 509783752704 count 1
> [26062.127298] 		ref#2: shared data backref parent 509351313408 count 1
> [26062.127299] 	item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183
> [26062.127303] 		extent refs 22 gen 1800674 flags 1
> [26062.127304] 		ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2
> [26062.127306] 		ref#1: shared data backref parent 508152889344 count 2
> [26062.127307] 		ref#2: shared data backref parent 508031139840 count 2
> [26062.127308] 		ref#3: shared data backref parent 493563478016 count 2
> [26062.127310] 		ref#4: shared data backref parent 493539983360 count 2
> [26062.127311] 		ref#5: shared data backref parent 492674990080 count 2
> [26062.127312] 		ref#6: shared data backref parent 475132166144 count 2
> [26062.127314] 		ref#7: shared data backref parent 471032348672 count 2
> [26062.127315] 		ref#8: shared data backref parent 436111769600 count 2
> [26062.127317] 		ref#9: shared data backref parent 883359744 count 2
> [26062.127318] 		ref#10: shared data backref parent 85082112 count 2
> [26062.127320] 	item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183
> [26062.127321] 		extent refs 33 gen 1800674 flags 1
> [26062.127322] 		ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3
> [26062.127324] 		ref#1: shared data backref parent 508152889344 count 3
> [26062.127325] 		ref#2: shared data backref parent 508031139840 count 3
> [26062.127327] 		ref#3: shared data backref parent 493563478016 count 3
> [26062.127328] 		ref#4: shared data backref parent 493539983360 count 3
> [26062.127329] 		ref#5: shared data backref parent 492674990080 count 3
> [26062.127331] 		ref#6: shared data backref parent 475132166144 count 3
> [26062.127333] 		ref#7: shared data backref parent 471032348672 count 3
> [26062.127334] 		ref#8: shared data backref parent 436111769600 count 3
> [26062.127335] 		ref#9: shared data backref parent 883359744 count 3
> [26062.127337] 		ref#10: shared data backref parent 85082112 count 3
> [26062.127338] 	item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92
> [26062.127340] 		extent refs 4 gen 2419085 flags 1
> [26062.127341] 		ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1
> [26062.127342] 		ref#1: shared data backref parent 494075035648 count 1
> [26062.127344] 		ref#2: shared data backref parent 470850240512 count 1
> [26062.127345] 		ref#3: shared data backref parent 437034745856 count 1
> [26062.127347] 	item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66
> [26062.127348] 		extent refs 2 gen 2368932 flags 1
> [26062.127349] 		ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1
> [26062.127351] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.127353] 	item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37
> [26062.127354] 		extent refs 1 gen 2369353 flags 1
> [26062.127355] 		ref#0: shared data backref parent 510165155840 count 1
> [26062.127356] 	item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66
> [26062.127358] 		extent refs 2 gen 2369168 flags 1
> [26062.127359] 		ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1
> [26062.127360] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.127362] 	item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53
> [26062.127364] 		extent refs 1 gen 2369305 flags 1
> [26062.127365] 		ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1
> [26062.127367] 	item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92
> [26062.127369] 		extent refs 4 gen 2419085 flags 1
> [26062.127370] 		ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1
> [26062.127372] 		ref#1: shared data backref parent 494075035648 count 1
> [26062.127374] 		ref#2: shared data backref parent 470850240512 count 1
> [26062.127375] 		ref#3: shared data backref parent 437034745856 count 1
> [26062.127377] 	item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66
> [26062.127378] 		extent refs 2 gen 2368955 flags 1
> [26062.127379] 		ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1
> [26062.127382] 		ref#1: shared data backref parent 437615230976 count 1
> [26062.127384] 	item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37
> [26062.127385] 		extent refs 1 gen 2369352 flags 1
> [26062.127386] 		ref#0: shared data backref parent 510178295808 count 1
> [26062.127388] 	item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37
> [26062.127390] 		extent refs 1 gen 2369352 flags 1
> [26062.127391] 		ref#0: shared data backref parent 510058790912 count 1
> [26062.127393] 	item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37
> [26062.127394] 		extent refs 1 gen 2369352 flags 1
> [26062.127395] 		ref#0: shared data backref parent 510059184128 count 1
> [26062.127398] 	item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53
> [26062.127399] 		extent refs 1 gen 2378552 flags 1
> [26062.127400] 		ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1
> [26062.127403] 	item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53
> [26062.127405] 		extent refs 1 gen 2378552 flags 1
> [26062.127406] 		ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1
> [26062.127408] 	item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53
> [26062.127409] 		extent refs 1 gen 2381741 flags 1
> [26062.127410] 		ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1
> [26062.127413] 	item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66
> [26062.127414] 		extent refs 2 gen 2368700 flags 1
> [26062.127415] 		ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1
> [26062.127416] 		ref#1: shared data backref parent 494105804800 count 1
> [26062.127419] 	item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66
> [26062.127420] 		extent refs 2 gen 2368700 flags 1
> [26062.127421] 		ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1
> [26062.127423] 		ref#1: shared data backref parent 494105804800 count 1
> [26062.127425] 	item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66
> [26062.127426] 		extent refs 2 gen 2368700 flags 1
> [26062.127428] 		ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1
> [26062.127430] 		ref#1: shared data backref parent 494105804800 count 1
> [26062.127432] 	item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183
> [26062.127434] 		extent refs 11 gen 1800699 flags 1
> [26062.127435] 		ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1
> [26062.127437] 		ref#1: shared data backref parent 508152987648 count 1
> [26062.127439] 		ref#2: shared data backref parent 508031172608 count 1
> [26062.127440] 		ref#3: shared data backref parent 493563822080 count 1
> [26062.127441] 		ref#4: shared data backref parent 493540130816 count 1
> [26062.127443] 		ref#5: shared data backref parent 492675170304 count 1
> [26062.127444] 		ref#6: shared data backref parent 475884584960 count 1
> [26062.127446] 		ref#7: shared data backref parent 471032496128 count 1
> [26062.127447] 		ref#8: shared data backref parent 436111884288 count 1
> [26062.127448] 		ref#9: shared data backref parent 883703808 count 1
> [26062.127450] 		ref#10: shared data backref parent 85671936 count 1
> [26062.127452] 	item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37
> [26062.127453] 		extent refs 1 gen 1444854 flags 1
> [26062.127454] 		ref#0: shared data backref parent 214908928 count 1
> [26062.127457] 	item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37
> [26062.127458] 		extent refs 1 gen 2426255 flags 1
> [26062.127459] 		ref#0: shared data backref parent 436271005696 count 1
> [26062.127461] 	item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183
> [26062.127463] 		extent refs 22 gen 1800674 flags 1
> [26062.127464] 		ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2
> [26062.127465] 		ref#1: shared data backref parent 508152987648 count 2
> [26062.127467] 		ref#2: shared data backref parent 508031172608 count 2
> [26062.127468] 		ref#3: shared data backref parent 493563822080 count 2
> [26062.127470] 		ref#4: shared data backref parent 493540130816 count 2
> [26062.127472] 		ref#5: shared data backref parent 492675170304 count 2
> [26062.127473] 		ref#6: shared data backref parent 475884584960 count 2
> [26062.127475] 		ref#7: shared data backref parent 471032496128 count 2
> [26062.127476] 		ref#8: shared data backref parent 436111884288 count 2
> [26062.127478] 		ref#9: shared data backref parent 883703808 count 2
> [26062.127480] 		ref#10: shared data backref parent 85671936 count 2
> [26062.127483] 	item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37
> [26062.127486] 		extent refs 1 gen 1444854 flags 1
> [26062.127488] 		ref#0: shared data backref parent 167198720 count 1
> [26062.127491] 	item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66
> [26062.127494] 		extent refs 2 gen 1400856 flags 1
> [26062.127496] 		ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1
> [26062.127498] 		ref#1: shared data backref parent 470954196992 count 1
> [26062.127501] 	item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37
> [26062.127503] 		extent refs 1 gen 1444854 flags 1
> [26062.127504] 		ref#0: shared data backref parent 93732864 count 1
> [26062.127507] 	item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53
> [26062.127510] 		extent refs 1 gen 2381741 flags 1
> [26062.127512] 		ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1
> [26062.127518] 	item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66
> [26062.127520] 		extent refs 2 gen 2368701 flags 1
> [26062.127521] 		ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1
> [26062.127523] 		ref#1: shared data backref parent 494107131904 count 1
> [26062.127526] 	item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183
> [26062.127527] 		extent refs 44 gen 1800674 flags 1
> [26062.127528] 		ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4
> [26062.127530] 		ref#1: shared data backref parent 508152987648 count 4
> [26062.127532] 		ref#2: shared data backref parent 508031172608 count 4
> [26062.127533] 		ref#3: shared data backref parent 493563822080 count 4
> [26062.127535] 		ref#4: shared data backref parent 493540130816 count 4
> [26062.127537] 		ref#5: shared data backref parent 492675170304 count 4
> [26062.127539] 		ref#6: shared data backref parent 475884584960 count 4
> [26062.127540] 		ref#7: shared data backref parent 471032496128 count 4
> [26062.127542] 		ref#8: shared data backref parent 436111884288 count 4
> [26062.127543] 		ref#9: shared data backref parent 883703808 count 4
> [26062.127545] 		ref#10: shared data backref parent 85671936 count 4
> [26062.127547] 	item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79
> [26062.127549] 		extent refs 3 gen 1800750 flags 1
> [26062.127552] 		ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1
> [26062.127554] 		ref#1: shared data backref parent 509783130112 count 1
> [26062.127557] 		ref#2: shared data backref parent 508544876544 count 1
> [26062.127560] 	item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37
> [26062.127561] 		extent refs 1 gen 1444854 flags 1
> [26062.127562] 		ref#0: shared data backref parent 93732864 count 1
> [26062.127565] 	item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37
> [26062.127566] 		extent refs 1 gen 1444854 flags 1
> [26062.127568] 		ref#0: shared data backref parent 214908928 count 1
> [26062.127570] 	item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37
> [26062.127572] 		extent refs 1 gen 1444854 flags 1
> [26062.127573] 		ref#0: shared data backref parent 93732864 count 1
> [26062.127577] 	item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24
> [26062.127580] 		block group used 834060288 chunk_objectid 256 flags 1
> [26062.127581] 	item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37
> [26062.127583] 		extent refs 1 gen 1444854 flags 1
> [26062.127584] 		ref#0: shared data backref parent 167198720 count 1
> [26062.127586] 	item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37
> [26062.127587] 		extent refs 1 gen 1444854 flags 1
> [26062.127588] 		ref#0: shared data backref parent 167198720 count 1
> [26062.127590] 	item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37
> [26062.127591] 		extent refs 1 gen 1444854 flags 1
> [26062.127592] 		ref#0: shared data backref parent 167198720 count 1
> [26062.127594] 	item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37
> [26062.127595] 		extent refs 1 gen 1444854 flags 1
> [26062.127596] 		ref#0: shared data backref parent 167198720 count 1
> [26062.127598] 	item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37
> [26062.127599] 		extent refs 1 gen 1444854 flags 1
> [26062.127600] 		ref#0: shared data backref parent 167198720 count 1
> [26062.127602] 	item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37
> [26062.127603] 		extent refs 1 gen 1444854 flags 1
> [26062.127604] 		ref#0: shared data backref parent 167198720 count 1
> [26062.127606] 	item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183
> [26062.127607] 		extent refs 11 gen 2409230 flags 1
> [26062.127608] 		ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1
> [26062.127610] 		ref#1: shared data backref parent 508153675776 count 1
> [26062.127611] 		ref#2: shared data backref parent 493566640128 count 1
> [26062.127613] 		ref#3: shared data backref parent 493540999168 count 1
> [26062.127614] 		ref#4: shared data backref parent 492676497408 count 1
> [26062.127616] 		ref#5: shared data backref parent 475884634112 count 1
> [26062.127617] 		ref#6: shared data backref parent 475691106304 count 1
> [26062.127619] 		ref#7: shared data backref parent 471034494976 count 1
> [26062.127620] 		ref#8: shared data backref parent 436113883136 count 1
> [26062.127622] 		ref#9: shared data backref parent 886407168 count 1
> [26062.127623] 		ref#10: shared data backref parent 89440256 count 1
> [26062.127625] 	item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170
> [26062.127626] 		extent refs 10 gen 2413323 flags 1
> [26062.127629] 		ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1
> [26062.127631] 		ref#1: shared data backref parent 508153675776 count 1
> [26062.127633] 		ref#2: shared data backref parent 493566640128 count 1
> [26062.127635] 		ref#3: shared data backref parent 493540999168 count 1
> [26062.127636] 		ref#4: shared data backref parent 492676497408 count 1
> [26062.127638] 		ref#5: shared data backref parent 475691106304 count 1
> [26062.127640] 		ref#6: shared data backref parent 471034494976 count 1
> [26062.127642] 		ref#7: shared data backref parent 436113883136 count 1
> [26062.127643] 		ref#8: shared data backref parent 886407168 count 1
> [26062.127645] 		ref#9: shared data backref parent 89440256 count 1
> [26062.127647] 	item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37
> [26062.127648] 		extent refs 1 gen 2407793 flags 1
> [26062.127649] 		ref#0: shared data backref parent 475884634112 count 1
> [26062.127653] 	item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183
> [26062.127655] 		extent refs 11 gen 1800729 flags 1
> [26062.127656] 		ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1
> [26062.127658] 		ref#1: shared data backref parent 508153069568 count 1
> [26062.127659] 		ref#2: shared data backref parent 508031270912 count 1
> [26062.127661] 		ref#3: shared data backref parent 493564215296 count 1
> [26062.127663] 		ref#4: shared data backref parent 493540098048 count 1
> [26062.127664] 		ref#5: shared data backref parent 492676038656 count 1
> [26062.127666] 		ref#6: shared data backref parent 475884142592 count 1
> [26062.127668] 		ref#7: shared data backref parent 471032840192 count 1
> [26062.127669] 		ref#8: shared data backref parent 436112146432 count 1
> [26062.127671] 		ref#9: shared data backref parent 883752960 count 1
> [26062.127674] 		ref#10: shared data backref parent 86654976 count 1
> [26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456  owner 73020573 offset 17039360
> [26062.127680] ------------[ cut here ]------------
> [26062.127682] BTRFS: Transaction aborted (error -2)
> [26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921
> [26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_h
 da_intel
> [26062.127733]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
> [26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
> [26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
> [26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921
> [26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df
> [26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282
> [26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007
> [26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550
> [26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
> [26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169
> [26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
> [26062.127815] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
> [26062.127817] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
> [26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [26062.127823] Call Trace:
> [26062.127832]  __btrfs_run_delayed_refs+0x750/0xc36
> [26062.127837]  ? __switch_to_asm+0x41/0x70
> [26062.127839]  ? __switch_to_asm+0x35/0x70
> [26062.127841]  ? __switch_to_asm+0x41/0x70
> [26062.127848]  ? __switch_to+0x13d/0x3d5
> [26062.127852]  btrfs_run_delayed_refs+0x5d/0x132
> [26062.127855]  btrfs_commit_transaction+0x55/0x7c8
> [26062.127858]  ? start_transaction+0x347/0x3cb
> [26062.127862]  transaction_kthread+0xc9/0x135
> [26062.127865]  ? btrfs_cleanup_transaction+0x403/0x403
> [26062.127869]  kthread+0xeb/0xf0
> [26062.127872]  ? kthread_create_worker_on_cpu+0x65/0x65
> [26062.127875]  ret_from_fork+0x35/0x40
> [26062.127879] ---[ end trace 4c1a6b3749a2f651 ]---
> [26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry
> [26062.127917] BTRFS info (device dm-2): forced readonly
> [26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entry

On Tue, Jul 17, 2018 at 05:24:51PM -0700, Marc MERLIN wrote:
> On Wed, Jul 18, 2018 at 08:05:51AM +0800, Qu Wenruo wrote:
> > No OOM triggers? That's a little strange.
> > Maybe it's related to how kernel handles memory over-commit?
>  
> Yes, I think you are correct.
> 
> > And for the hang, I think it's related to some memory allocation failure
> > and error handler just didn't handle it well, so it's causing deadlock
> > for certain page.
> 
> That indeed matches what I'm seeing.
> 
> > ENOMEM handling is pretty common but hardly verified, so it's not that
> > strange, but we must locate the problem.
> 
> I seem to be getting deadlocks in the kernel, so I'm hoping that at least
> it's checked there, but maybe not?
> 
> > In my system, at least I'm not using btrfs as root fs, and for the
> > memory eating program I normally ensure it's eating all the memory +
> > swap, so OOM killer is always triggered, maybe that's the cause.
> > 
> > So in your case, maybe it's btrfs not really taking up all memory, thus
> > OOM killer not triggered.
> 
> Correct, the swap is not used.
> 
> > Any kernel dmesg about OOM killer triggered?
>  
> Nothing at all. It never gets triggered.
> 
> > > Here is my system when it virtually died:
> > > ER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> > > root     31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49   1:35 ./btrfs check /dev/mapper/dshelf2
> 
> See how btrs was taking 29GB in that ps output (that's before it takes
> everything and I can't even type ps anymore)
> Note that VSZ is almost equal to RSS. Nothing gets swapped.
> 
> Then see free output:
> 
> > >              total       used       free     shared    buffers     cached
> > > Mem:      32643788   32180100     463688          0      44664     119508
> > > -/+ buffers/cache:   32015928     627860
> > > Swap:     15616764     443676   15173088
> > 
> > For swap, it looks like only some other program's memory is swapped out,
> > not btrfs'.
> 
> That's exactly correct. btrfs check never goes to swap, I'm not sure why,
> and because there is virtual memory free, maybe that's why OOM does not
> trigger?
> So I guess I can probably "fix" my problem by removing swap, but ultimately
> it would be useful to know why memory taken by btrfs check does not end up
> in swap.
> 
> > And unfortunately, I'm not so familiar with OOM/MM code outside of
> > filesystem.
> > Any help from other experienced developers would definitely help to
> > solve why memory of 'btrfs check' is not swapped out or why OOM killer
> > is not triggered.
> 
> Do you have someone from linux-vm you might be able to ask, or should we Cc
> this thread there?
> 
> Thanks,
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/  

On Tue, Jul 17, 2018 at 01:59:05PM -0700, Marc MERLIN wrote:
> Ok, I did more testing. Qu is right that btrfs check does not crash the kernel.
> It just takes all the memory until linux hangs everywhere, and somehow (no idea why) 
> the OOM killer never triggers.
> Details below:
> 
> On Tue, Jul 17, 2018 at 01:32:57PM -0700, Marc MERLIN wrote:
> > Here is what I got when the system was not doing well (it took minutes to run):
> > 
> >              total       used       free     shared    buffers     cached
> > Mem:      32643788   32070952     572836          0     102160    4378772
> > -/+ buffers/cache:   27590020    5053768
> > Swap:     15616764     973596   14643168
> 
> ok, the reason it was not that close to 0 was due to /dev/shm it seems.
> I cleared that, and now I can get it to go to near 0 again.
> I'm wrong about the system being fully crashed, it's not, it's just very
> close to being hung.
> I can type killall -9 btrfs in the serial console and wait a few minutes.
> The system eventually recovers, but it's impossible to fix anything via ssh 
> apparently because networking does not get to run when I'm in this state.
> 
> I'm not sure why my system reproduces this easy while Qu's system does not, 
> but Qu was right that the kernel is not dead and that it's merely a problem of userspace
> taking all the RAM and somehow not being killed by OOM
> 
> I checked the PID and don't see why it's not being killed:
> gargamel:/proc/31006# grep . oom*
> oom_adj:0
> oom_score:221   << this increases a lot, but OOM never kills it
> oom_score_adj:0
> 
> I have these variables:
> /proc/sys/vm/oom_dump_tasks:1
> /proc/sys/vm/oom_kill_allocating_task:0
> /proc/sys/vm/overcommit_kbytes:0
> /proc/sys/vm/overcommit_memory:0
> /proc/sys/vm/overcommit_ratio:50  << is this bad (seems default)
> 
> Here is my system when it virtually died:
> ER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> root     31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49   1:35 ./btrfs check /dev/mapper/dshelf2
> 
>              total       used       free     shared    buffers     cached
> Mem:      32643788   32180100     463688          0      44664     119508
> -/+ buffers/cache:   32015928     627860
> Swap:     15616764     443676   15173088
> 
> MemTotal:       32643788 kB
> MemFree:          463440 kB
> MemAvailable:      44864 kB
> Buffers:           44664 kB
> Cached:           120360 kB
> SwapCached:        87064 kB
> Active:         30381404 kB
> Inactive:         585952 kB
> Active(anon):   30334696 kB
> Inactive(anon):   474624 kB
> Active(file):      46708 kB
> Inactive(file):   111328 kB
> Unevictable:        5616 kB
> Mlocked:            5616 kB
> SwapTotal:      15616764 kB
> SwapFree:       15173088 kB
> Dirty:              1636 kB
> Writeback:             4 kB
> AnonPages:      30734240 kB
> Mapped:            67236 kB
> Shmem:              3036 kB
> Slab:             267884 kB
> SReclaimable:      51528 kB
> SUnreclaim:       216356 kB
> KernelStack:       10144 kB
> PageTables:        69284 kB
> NFS_Unstable:          0 kB
> Bounce:                0 kB
> WritebackTmp:          0 kB
> CommitLimit:    31938656 kB
> Committed_AS:   32865492 kB
> VmallocTotal:   34359738367 kB
> VmallocUsed:           0 kB
> VmallocChunk:          0 kB
> HardwareCorrupted:     0 kB
> AnonHugePages:         0 kB
> ShmemHugePages:        0 kB
> ShmemPmdMapped:        0 kB
> CmaTotal:          16384 kB
> CmaFree:               0 kB
> HugePages_Total:       0
> HugePages_Free:        0
> HugePages_Rsvd:        0
> HugePages_Surp:        0
> Hugepagesize:       2048 kB
> Hugetlb:               0 kB
> DirectMap4k:      560404 kB
> DirectMap2M:    32692224 kB
> 
> 
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/  


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
  2019-10-19  3:07         ` Marc MERLIN
@ 2019-10-23  0:55           ` Marc MERLIN
  2019-10-26  3:36             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2019-10-23  0:55 UTC (permalink / raw)
  To: Linux BTRFS Mailinglist, Qu Wenruo, Su Yue

On Fri, Oct 18, 2019 at 08:07:28PM -0700, Marc MERLIN wrote:
> Ok, so before blowing the filesystem away after it was apparently badly
> damaged by a suspend to disk, I tried check --repair and I hit an
> infinite loop.
> 
> Let me know if you'd like anything off the FS before I delete it.

I heard nothing back, so I deleted the FS and restored from backup.

But now I'm scared of ever doing a suspend to disk again.
Could someone please look at the logs and give me some idea of what
happened, if at all possible?

Non recoverable data corruption on my laptop when I travel and
backups/restores are complicated, is a bit unnerving...

Thanks,
Marc

> Thanks,
> Marc
> 
> enabling repair mode
> repair mode will force to clear out log tree, are you sure? [y/N]: y
> Checking filesystem on /dev/mapper/pool1
> UUID: fda628bc-1ca4-49c5-91c2-4260fe967a23
> checking extents
> Backref 415334400 parent 36028797198598144 not referenced back 0x5648ef1870e0
> Backref 415334400 parent 179634176 root 179634176 not found in extent tree
> Incorrect global backref count on 415334400 found 2 wanted 1
> backpointer mismatch on [415334400 16384]
> repair deleting extent record: key 415334400 169 0
> adding new tree backref on start 415334400 len 16384 parent 179634176 root 179634176
> Repaired extent references for 415334400
> ref mismatch on [101995261952 4096] extent item 36028797018963969, found 1
> repair deleting extent record: key 101995261952 168 4096
> adding new data backref on 101995261952 root 456 owner 74455677 offset 64892928 found 1
> Repaired extent references for 101995261952
> Incorrect local backref count on 458640384000 root 456 owner 81409181 offset 17039360 found 0 wanted 1 back 0x5648eefd3d10
> Backref disk bytenr does not match extent record, bytenr=458640384000, ref bytenr=0
> Backref 458640384000 root 456 owner 73020573 offset 17039360 num_refs 0 not found in extent tree
> Incorrect local backref count on 458640384000 root 456 owner 73020573 offset 17039360 found 1 wanted 0 back 0x5648b32a9600
> backpointer mismatch on [458640384000 86016]
> repair deleting extent record: key 458640384000 168 86016
> adding new data backref on 458640384000 parent 438017720320 owner 0 offset 0 found 1
> adding new data backref on 458640384000 root 456 owner 73020573 offset 17039360 found 1
> Repaired extent references for 458640384000
> Fixed 0 roots.
> checking free space cache
> cache and super generation don't match, space cache will be invalidated
> checking fs roots
> Deleting bad dir index [10138517,96,436945] root 456
> Deleting bad dir index [10138518,96,646273] root 456
> Deleting bad dir index [10138517,96,437016] root 456
> Deleting bad dir index [10138518,96,808999] root 456
> Deleting bad dir index [10215134,96,149427] root 456
> Deleting bad dir index [10240541,96,268037] root 456
> Deleting bad dir index [10138517,96,540247] root 456
> Deleting bad dir index [10138518,96,825234] root 456
> Deleting bad dir index [10138517,96,736673] root 456
> Deleting bad dir index [10138518,96,1118221] root 456
> Deleting bad dir index [10240541,96,439703] root 456
> Deleting bad dir index [10138517,96,752282] root 456
> root 456 inode 75431563 errors 100, file extent discount
> Found file extent holes:
> 	start: 4096, len: 4096
> root 456 inode 75431568 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 1638400
> root 456 inode 75431583 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 147456
> root 456 inode 75431585 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 208896
> root 456 inode 75431591 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 2523136
> root 456 inode 75431730 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 208896
> root 456 inode 75431744 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 2084864
> root 456 inode 75431751 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 172032
> root 456 inode 75431756 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 8192
> root 456 inode 75431760 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 12288
> root 456 inode 75431765 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 32768
> root 456 inode 75431773 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 90112
> Fixed discount file extents for inode: 75432421 in root: 456
> Fixed discount file extents for inode: 75432429 in root: 456
> root 456 inode 75432429 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 4096
> Fixed discount file extents for inode: 75432430 in root: 456
> Fixed discount file extents for inode: 75432432 in root: 456
> Fixed discount file extents for inode: 75432433 in root: 456
> root 456 inode 75432433 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 4096
> root 456 inode 75432434 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 724992
> Fixed discount file extents for inode: 75432436 in root: 456
> Fixed discount file extents for inode: 75432437 in root: 456
> root 456 inode 75432437 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 4096
> Fixed discount file extents for inode: 75432438 in root: 456
> Fixed discount file extents for inode: 75432441 in root: 456
> root 456 inode 75432441 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 4096
> Fixed discount file extents for inode: 75432444 in root: 456
> root 456 inode 75432444 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 4096
> Fixed discount file extents for inode: 75432445 in root: 456
> Fixed discount file extents for inode: 75432447 in root: 456
> Fixed discount file extents for inode: 75432448 in root: 456
> Fixed discount file extents for inode: 75432454 in root: 456
> Fixed discount file extents for inode: 75432457 in root: 456
> root 456 inode 75432457 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 4096
> root 456 inode 75432471 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 20480
> Fixed discount file extents for inode: 75432473 in root: 456
> root 456 inode 75432473 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 20480
> Fixed discount file extents for inode: 75432482 in root: 456
> Fixed discount file extents for inode: 75432483 in root: 456
> Fixed discount file extents for inode: 75432484 in root: 456
> Fixed discount file extents for inode: 75432487 in root: 456
> Fixed discount file extents for inode: 75432489 in root: 456
> Fixed discount file extents for inode: 75432494 in root: 456
> Fixed discount file extents for inode: 75432611 in root: 456
> Fixed discount file extents for inode: 75432622 in root: 456
> Fixed discount file extents for inode: 75432626 in root: 456
> Fixed discount file extents for inode: 75432628 in root: 456
> Fixed discount file extents for inode: 75432636 in root: 456
> Fixed discount file extents for inode: 75432637 in root: 456
> root 456 inode 75432637 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 4096
> Fixed discount file extents for inode: 75432643 in root: 456
> root 456 inode 75432643 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 135168
> Fixed discount file extents for inode: 75432650 in root: 456
> root 456 inode 75432650 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 4096
> Fixed discount file extents for inode: 75432656 in root: 456
> root 456 inode 75432656 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 4096
> Fixed discount file extents for inode: 75432657 in root: 456
> root 456 inode 75432657 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 4096
> Fixed discount file extents for inode: 75432659 in root: 456
> Fixed discount file extents for inode: 75432662 in root: 456
> root 456 inode 75432662 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 8192
> Fixed discount file extents for inode: 75432663 in root: 456
> root 456 inode 75432663 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 8192
> Fixed discount file extents for inode: 75432664 in root: 456
> root 456 inode 75432664 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 8192
> Fixed discount file extents for inode: 75432674 in root: 456
> root 456 inode 75432674 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 8192
> Fixed discount file extents for inode: 75432692 in root: 456
> root 456 inode 75432692 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 4096
> Fixed discount file extents for inode: 75432708 in root: 456
> Fixed discount file extents for inode: 75432709 in root: 456
> root 456 inode 75432709 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 4096
> Fixed discount file extents for inode: 75432712 in root: 456
> Fixed discount file extents for inode: 75432717 in root: 456
> Fixed discount file extents for inode: 75432720 in root: 456
> Fixed discount file extents for inode: 75432725 in root: 456
> Fixed discount file extents for inode: 75432731 in root: 456
> Fixed discount file extents for inode: 75432732 in root: 456
> Fixed discount file extents for inode: 75432749 in root: 456
> Fixed discount file extents for inode: 75432762 in root: 456
> Fixed discount file extents for inode: 75432765 in root: 456
> Fixed discount file extents for inode: 75432783 in root: 456
> Fixed discount file extents for inode: 75432784 in root: 456
> Fixed discount file extents for inode: 75432785 in root: 456
> root 456 inode 75432785 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 8192
> Fixed discount file extents for inode: 75432786 in root: 456
> root 456 inode 75432786 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 8192
> Fixed discount file extents for inode: 75432796 in root: 456
> root 456 inode 75432796 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 8192
> Fixed discount file extents for inode: 75432799 in root: 456
> root 456 inode 75432799 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 8192
> Fixed discount file extents for inode: 75432801 in root: 456
> root 456 inode 75432801 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 4096
> Fixed discount file extents for inode: 75432807 in root: 456
> Fixed discount file extents for inode: 75432817 in root: 456
> Fixed discount file extents for inode: 75432829 in root: 456
> Fixed discount file extents for inode: 75432860 in root: 456
> root 456 inode 75432860 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 4096
> Fixed discount file extents for inode: 75432862 in root: 456
> Fixed discount file extents for inode: 75432863 in root: 456
> Fixed discount file extents for inode: 75432869 in root: 456
> Fixed discount file extents for inode: 75432870 in root: 456
> Fixed discount file extents for inode: 75432871 in root: 456
> Fixed discount file extents for inode: 75432872 in root: 456
> Fixed discount file extents for inode: 75432875 in root: 456
> Fixed discount file extents for inode: 75432877 in root: 456
> Fixed discount file extents for inode: 75432882 in root: 456
> Fixed discount file extents for inode: 75432883 in root: 456
> Fixed discount file extents for inode: 75432893 in root: 456
> Fixed discount file extents for inode: 75432894 in root: 456
> Fixed discount file extents for inode: 75432897 in root: 456
> Fixed discount file extents for inode: 75432899 in root: 456
> Fixed discount file extents for inode: 75432900 in root: 456
> Fixed discount file extents for inode: 75432905 in root: 456
> Fixed discount file extents for inode: 75432906 in root: 456
> Fixed discount file extents for inode: 75432916 in root: 456
> Fixed discount file extents for inode: 75432917 in root: 456
> Fixed discount file extents for inode: 75432919 in root: 456
> Fixed discount file extents for inode: 75432920 in root: 456
> Fixed discount file extents for inode: 75432923 in root: 456
> Fixed discount file extents for inode: 75432942 in root: 456
> Fixed discount file extents for inode: 75432944 in root: 456
> Fixed discount file extents for inode: 75432948 in root: 456
> root 456 inode 75432948 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 8192
> Fixed discount file extents for inode: 75432949 in root: 456
> root 456 inode 75432949 errors 100, file extent discount
> Found file extent holes:
> 	start: 0, len: 8192
> and it loops forever on 456
> 
> On Thu, Oct 17, 2019 at 07:56:04PM -0700, Marc MERLIN wrote:
> > This happened almost after a resume from suspend to disk.
> > First corruption and read only I got a very long time.
> > 
> > Could they be related?
> > 
> > [26062.126505] ------------[ cut here ]------------
> > [26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
> > [26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd
 _hda_intel
> > [26062.126561]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
> > [26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
> > [26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
> > [26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921
> > [26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41
> > [26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246
> > [26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000
> > [26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558
> > [26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
> > [26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169
> > [26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
> > [26062.126625] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
> > [26062.126627] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
> > [26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > [26062.126636] Call Trace:
> > [26062.126647]  __btrfs_run_delayed_refs+0x750/0xc36
> > [26062.126653]  ? __switch_to_asm+0x41/0x70
> > [26062.126655]  ? __switch_to_asm+0x35/0x70
> > [26062.126658]  ? __switch_to_asm+0x41/0x70
> > [26062.126662]  ? __switch_to+0x13d/0x3d5
> > [26062.126668]  btrfs_run_delayed_refs+0x5d/0x132
> > [26062.126672]  btrfs_commit_transaction+0x55/0x7c8
> > [26062.126676]  ? start_transaction+0x347/0x3cb
> > [26062.126679]  transaction_kthread+0xc9/0x135
> > [26062.126683]  ? btrfs_cleanup_transaction+0x403/0x403
> > [26062.126688]  kthread+0xeb/0xf0
> > [26062.126692]  ? kthread_create_worker_on_cpu+0x65/0x65
> > [26062.126695]  ret_from_fork+0x35/0x40
> > [26062.126698] ---[ end trace 4c1a6b3749a2f650 ]---
> > [26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2
> > [26062.126706] 	item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66
> > [26062.126708] 		extent refs 2 gen 2369265 flags 1
> > [26062.126709] 		ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1
> > [26062.126711] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.126714] 	item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66
> > [26062.126715] 		extent refs 2 gen 2369025 flags 1
> > [26062.126716] 		ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1
> > [26062.126718] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.126720] 	item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183
> > [26062.126722] 		extent refs 11 gen 1800715 flags 1
> > [26062.126722] 		ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1
> > [26062.126724] 		ref#1: shared data backref parent 508153839616 count 1
> > [26062.126726] 		ref#2: shared data backref parent 493541244928 count 1
> > [26062.126727] 		ref#3: shared data backref parent 492677332992 count 1
> > [26062.126728] 		ref#4: shared data backref parent 492328566784 count 1
> > [26062.126730] 		ref#5: shared data backref parent 492245188608 count 1
> > [26062.126731] 		ref#6: shared data backref parent 475885109248 count 1
> > [26062.126732] 		ref#7: shared data backref parent 471036608512 count 1
> > [26062.126733] 		ref#8: shared data backref parent 436114243584 count 1
> > [26062.126735] 		ref#9: shared data backref parent 886718464 count 1
> > [26062.126736] 		ref#10: shared data backref parent 90259456 count 1
> > [26062.126738] 	item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66
> > [26062.126740] 		extent refs 2 gen 2420471 flags 1
> > [26062.126741] 		ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1
> > [26062.126743] 		ref#1: shared data backref parent 41020112896 count 1
> > [26062.126745] 	item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53
> > [26062.126746] 		extent refs 1 gen 2422941 flags 1
> > [26062.126747] 		ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1
> > [26062.126749] 	item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53
> > [26062.126750] 		extent refs 1 gen 1444854 flags 1
> > [26062.126751] 		ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1
> > [26062.126753] 	item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37
> > [26062.126754] 		extent refs 1 gen 1444854 flags 1
> > [26062.126755] 		ref#0: shared data backref parent 214892544 count 1
> > [26062.126757] 	item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37
> > [26062.126758] 		extent refs 1 gen 1444854 flags 1
> > [26062.126759] 		ref#0: shared data backref parent 102547456 count 1
> > [26062.126761] 	item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66
> > [26062.126762] 		extent refs 2 gen 2422644 flags 1
> > [26062.126763] 		ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1
> > [26062.126765] 		ref#1: shared data backref parent 470658482176 count 1
> > [26062.126768] 	item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66
> > [26062.126769] 		extent refs 2 gen 2369217 flags 1
> > [26062.126770] 		ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1
> > [26062.126772] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.126774] 	item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66
> > [26062.126775] 		extent refs 2 gen 2369242 flags 1
> > [26062.126776] 		ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1
> > [26062.126778] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.126780] 	item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53
> > [26062.126781] 		extent refs 1 gen 2369305 flags 1
> > [26062.126782] 		ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1
> > [26062.126784] 	item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66
> > [26062.126786] 		extent refs 2 gen 2369052 flags 1
> > [26062.126787] 		ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1
> > [26062.126788] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.126790] 	item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53
> > [26062.126792] 		extent refs 1 gen 2369305 flags 1
> > [26062.126793] 		ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1
> > [26062.126795] 	item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66
> > [26062.126796] 		extent refs 2 gen 2368754 flags 1
> > [26062.126797] 		ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1
> > [26062.126799] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.126801] 	item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183
> > [26062.126802] 		extent refs 11 gen 1800682 flags 1
> > [26062.126803] 		ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1
> > [26062.126805] 		ref#1: shared data backref parent 508153069568 count 1
> > [26062.126806] 		ref#2: shared data backref parent 508031270912 count 1
> > [26062.126808] 		ref#3: shared data backref parent 493564215296 count 1
> > [26062.126809] 		ref#4: shared data backref parent 493540098048 count 1
> > [26062.126811] 		ref#5: shared data backref parent 492676038656 count 1
> > [26062.126813] 		ref#6: shared data backref parent 475884142592 count 1
> > [26062.126814] 		ref#7: shared data backref parent 471032840192 count 1
> > [26062.126816] 		ref#8: shared data backref parent 436112146432 count 1
> > [26062.126817] 		ref#9: shared data backref parent 883752960 count 1
> > [26062.126818] 		ref#10: shared data backref parent 86654976 count 1
> > [26062.126820] 	item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50
> > [26062.126822] 		extent refs 2 gen 2369347 flags 1
> > [26062.126823] 		ref#0: shared data backref parent 492981370880 count 1
> > [26062.126824] 		ref#1: shared data backref parent 438017720320 count 1
> > [26062.126826] 	item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50
> > [26062.126827] 		extent refs 2 gen 2369347 flags 1
> > [26062.126828] 		ref#0: shared data backref parent 492981370880 count 1
> > [26062.126830] 		ref#1: shared data backref parent 438017720320 count 1
> > [26062.126832] 	item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50
> > [26062.126833] 		extent refs 2 gen 2369347 flags 1
> > [26062.126834] 		ref#0: shared data backref parent 492981370880 count 1
> > [26062.126836] 		ref#1: shared data backref parent 438017720320 count 1
> > [26062.126839] 	item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37
> > [26062.126840] 		extent refs 1 gen 2369352 flags 1
> > [26062.126841] 		ref#0: shared data backref parent 510058790912 count 1
> > [26062.126843] 	item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37
> > [26062.126845] 		extent refs 1 gen 2369352 flags 1
> > [26062.126845] 		ref#0: shared data backref parent 510058790912 count 1
> > [26062.126848] 	item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37
> > [26062.126849] 		extent refs 1 gen 2369352 flags 1
> > [26062.126850] 		ref#0: shared data backref parent 510058790912 count 1
> > [26062.126852] 	item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50
> > [26062.126853] 		extent refs 2 gen 2369347 flags 1
> > [26062.126854] 		ref#0: shared data backref parent 492981370880 count 1
> > [26062.126856] 		ref#1: shared data backref parent 438017720320 count 1
> > [26062.126858] 	item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50
> > [26062.126859] 		extent refs 2 gen 2369347 flags 1
> > [26062.126860] 		ref#0: shared data backref parent 492981370880 count 1
> > [26062.126862] 		ref#1: shared data backref parent 438017720320 count 1
> > [26062.126864] 	item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50
> > [26062.126865] 		extent refs 2 gen 2369347 flags 1
> > [26062.126866] 		ref#0: shared data backref parent 492981370880 count 1
> > [26062.126868] 		ref#1: shared data backref parent 438017720320 count 1
> > [26062.126870] 	item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50
> > [26062.126871] 		extent refs 2 gen 2369347 flags 1
> > [26062.126873] 		ref#0: shared data backref parent 492981370880 count 1
> > [26062.126874] 		ref#1: shared data backref parent 438017720320 count 1
> > [26062.126876] 	item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50
> > [26062.126877] 		extent refs 2 gen 2369347 flags 1
> > [26062.126878] 		ref#0: shared data backref parent 492981370880 count 1
> > [26062.126880] 		ref#1: shared data backref parent 438017720320 count 1
> > [26062.126882] 	item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50
> > [26062.126883] 		extent refs 2 gen 2369347 flags 1
> > [26062.126884] 		ref#0: shared data backref parent 492981370880 count 1
> > [26062.126885] 		ref#1: shared data backref parent 438017720320 count 1
> > [26062.126888] 	item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50
> > [26062.126889] 		extent refs 2 gen 2369347 flags 1
> > [26062.126890] 		ref#0: shared data backref parent 492981370880 count 1
> > [26062.126891] 		ref#1: shared data backref parent 438017720320 count 1
> > [26062.126893] 	item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79
> > [26062.126894] 		extent refs 3 gen 2369347 flags 1
> > [26062.126895] 		ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1
> > [26062.126897] 		ref#1: shared data backref parent 492981370880 count 1
> > [26062.126898] 		ref#2: shared data backref parent 438017720320 count 1
> > [26062.126900] 	item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66
> > [26062.126902] 		extent refs 2 gen 2369347 flags 1
> > [26062.126903] 		ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1
> > [26062.126904] 		ref#1: shared data backref parent 438017720320 count 1
> > [26062.126906] 	item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37
> > [26062.126907] 		extent refs 1 gen 2369352 flags 1
> > [26062.126908] 		ref#0: shared data backref parent 510058790912 count 1
> > [26062.126910] 	item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79
> > [26062.126912] 		extent refs 6 gen 1800750 flags 1
> > [26062.126913] 		ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2
> > [26062.126915] 		ref#1: shared data backref parent 509782933504 count 2
> > [26062.126917] 		ref#2: shared data backref parent 508543795200 count 2
> > [26062.126919] 	item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66
> > [26062.126920] 		extent refs 2 gen 2369075 flags 1
> > [26062.126921] 		ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1
> > [26062.126923] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.126925] 	item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79
> > [26062.126926] 		extent refs 3 gen 1800733 flags 1
> > [26062.126927] 		ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1
> > [26062.126929] 		ref#1: shared data backref parent 509783719936 count 1
> > [26062.126930] 		ref#2: shared data backref parent 509351231488 count 1
> > [26062.126932] 	item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66
> > [26062.126934] 		extent refs 2 gen 2420471 flags 1
> > [26062.126934] 		ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1
> > [26062.126936] 		ref#1: shared data backref parent 41020112896 count 1
> > [26062.126938] 	item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66
> > [26062.126939] 		extent refs 2 gen 2420471 flags 1
> > [26062.126940] 		ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1
> > [26062.126943] 		ref#1: shared data backref parent 41020112896 count 1
> > [26062.126945] 	item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79
> > [26062.126946] 		extent refs 3 gen 1800733 flags 1
> > [26062.126947] 		ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1
> > [26062.126949] 		ref#1: shared data backref parent 509783719936 count 1
> > [26062.126950] 		ref#2: shared data backref parent 509351231488 count 1
> > [26062.126953] 	item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66
> > [26062.126954] 		extent refs 2 gen 2420502 flags 1
> > [26062.126955] 		ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1
> > [26062.126957] 		ref#1: shared data backref parent 492274728960 count 1
> > [26062.126959] 	item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66
> > [26062.126961] 		extent refs 2 gen 2420478 flags 1
> > [26062.126962] 		ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1
> > [26062.126964] 		ref#1: shared data backref parent 41020112896 count 1
> > [26062.126966] 	item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37
> > [26062.126967] 		extent refs 1 gen 1444854 flags 1
> > [26062.126968] 		ref#0: shared data backref parent 214908928 count 1
> > [26062.126970] 	item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53
> > [26062.126971] 		extent refs 1 gen 2376479 flags 1
> > [26062.126972] 		ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1
> > [26062.126975] 	item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53
> > [26062.126976] 		extent refs 1 gen 1444854 flags 1
> > [26062.126977] 		ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1
> > [26062.126979] 	item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37
> > [26062.126981] 		extent refs 1 gen 1444854 flags 1
> > [26062.126982] 		ref#0: shared data backref parent 93732864 count 1
> > [26062.126984] 	item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37
> > [26062.126985] 		extent refs 1 gen 1444854 flags 1
> > [26062.126986] 		ref#0: shared data backref parent 93732864 count 1
> > [26062.126988] 	item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37
> > [26062.126990] 		extent refs 1 gen 1444854 flags 1
> > [26062.126991] 		ref#0: shared data backref parent 93732864 count 1
> > [26062.126993] 	item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37
> > [26062.126994] 		extent refs 1 gen 1444854 flags 1
> > [26062.126995] 		ref#0: shared data backref parent 214908928 count 1
> > [26062.126997] 	item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92
> > [26062.126998] 		extent refs 4 gen 2419085 flags 1
> > [26062.126999] 		ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1
> > [26062.127001] 		ref#1: shared data backref parent 494075035648 count 1
> > [26062.127002] 		ref#2: shared data backref parent 470850240512 count 1
> > [26062.127004] 		ref#3: shared data backref parent 437034745856 count 1
> > [26062.127006] 	item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79
> > [26062.127008] 		extent refs 3 gen 1800733 flags 1
> > [26062.127009] 		ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1
> > [26062.127011] 		ref#1: shared data backref parent 509783719936 count 1
> > [26062.127013] 		ref#2: shared data backref parent 509351231488 count 1
> > [26062.127015] 	item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66
> > [26062.127016] 		extent refs 2 gen 2368859 flags 1
> > [26062.127017] 		ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1
> > [26062.127019] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.127021] 	item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53
> > [26062.127023] 		extent refs 1 gen 2376479 flags 1
> > [26062.127024] 		ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1
> > [26062.127026] 	item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183
> > [26062.127028] 		extent refs 11 gen 1800674 flags 1
> > [26062.127029] 		ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1
> > [26062.127031] 		ref#1: shared data backref parent 508153118720 count 1
> > [26062.127032] 		ref#2: shared data backref parent 508031533056 count 1
> > [26062.127034] 		ref#3: shared data backref parent 493564477440 count 1
> > [26062.127036] 		ref#4: shared data backref parent 493540179968 count 1
> > [26062.127038] 		ref#5: shared data backref parent 492675366912 count 1
> > [26062.127039] 		ref#6: shared data backref parent 475132116992 count 1
> > [26062.127041] 		ref#7: shared data backref parent 471032922112 count 1
> > [26062.127042] 		ref#8: shared data backref parent 436112244736 count 1
> > [26062.127044] 		ref#9: shared data backref parent 883998720 count 1
> > [26062.127046] 		ref#10: shared data backref parent 87015424 count 1
> > [26062.127048] 	item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37
> > [26062.127050] 		extent refs 1 gen 2408131 flags 1
> > [26062.127052] 		ref#0: shared data backref parent 509278633984 count 1
> > [26062.127055] 	item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183
> > [26062.127056] 		extent refs 11 gen 1800715 flags 1
> > [26062.127058] 		ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1
> > [26062.127059] 		ref#1: shared data backref parent 508152889344 count 1
> > [26062.127061] 		ref#2: shared data backref parent 508031139840 count 1
> > [26062.127062] 		ref#3: shared data backref parent 493563478016 count 1
> > [26062.127065] 		ref#4: shared data backref parent 493539983360 count 1
> > [26062.127066] 		ref#5: shared data backref parent 492674990080 count 1
> > [26062.127068] 		ref#6: shared data backref parent 475132166144 count 1
> > [26062.127070] 		ref#7: shared data backref parent 471032348672 count 1
> > [26062.127071] 		ref#8: shared data backref parent 436111769600 count 1
> > [26062.127073] 		ref#9: shared data backref parent 883359744 count 1
> > [26062.127074] 		ref#10: shared data backref parent 85082112 count 1
> > [26062.127077] 	item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92
> > [26062.127078] 		extent refs 4 gen 2419085 flags 1
> > [26062.127079] 		ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1
> > [26062.127085] 		ref#1: shared data backref parent 494075035648 count 1
> > [26062.127088] 		ref#2: shared data backref parent 470850240512 count 1
> > [26062.127089] 		ref#3: shared data backref parent 437034745856 count 1
> > [26062.127093] 	item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66
> > [26062.127095] 		extent refs 2 gen 2421893 flags 1
> > [26062.127096] 		ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1
> > [26062.127099] 		ref#1: shared data backref parent 493703069696 count 1
> > [26062.127102] 	item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53
> > [26062.127104] 		extent refs 1 gen 1444854 flags 1
> > [26062.127105] 		ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1
> > [26062.127109] 	item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66
> > [26062.127111] 		extent refs 2 gen 2368700 flags 1
> > [26062.127112] 		ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1
> > [26062.127114] 		ref#1: shared data backref parent 494105804800 count 1
> > [26062.127118] 	item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37
> > [26062.127120] 		extent refs 1 gen 1444854 flags 1
> > [26062.127121] 		ref#0: shared data backref parent 93732864 count 1
> > [26062.127125] 	item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37
> > [26062.127127] 		extent refs 1 gen 1444854 flags 1
> > [26062.127128] 		ref#0: shared data backref parent 214433792 count 1
> > [26062.127131] 	item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37
> > [26062.127132] 		extent refs 1 gen 1444854 flags 1
> > [26062.127133] 		ref#0: shared data backref parent 214908928 count 1
> > [26062.127136] 	item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79
> > [26062.127138] 		extent refs 3 gen 1800750 flags 1
> > [26062.127139] 		ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1
> > [26062.127141] 		ref#1: shared data backref parent 509783556096 count 1
> > [26062.127143] 		ref#2: shared data backref parent 508544974848 count 1
> > [26062.127147] 	item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66
> > [26062.127148] 		extent refs 2 gen 2368707 flags 1
> > [26062.127149] 		ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1
> > [26062.127151] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.127153] 	item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183
> > [26062.127155] 		extent refs 11 gen 1800674 flags 1
> > [26062.127156] 		ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1
> > [26062.127157] 		ref#1: shared data backref parent 508152905728 count 1
> > [26062.127159] 		ref#2: shared data backref parent 508031057920 count 1
> > [26062.127160] 		ref#3: shared data backref parent 493563166720 count 1
> > [26062.127161] 		ref#4: shared data backref parent 493539934208 count 1
> > [26062.127163] 		ref#5: shared data backref parent 492674908160 count 1
> > [26062.127164] 		ref#6: shared data backref parent 475884208128 count 1
> > [26062.127165] 		ref#7: shared data backref parent 471032266752 count 1
> > [26062.127167] 		ref#8: shared data backref parent 436111720448 count 1
> > [26062.127168] 		ref#9: shared data backref parent 883179520 count 1
> > [26062.127170] 		ref#10: shared data backref parent 83017728 count 1
> > [26062.127172] 	item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37
> > [26062.127173] 		extent refs 1 gen 1444854 flags 1
> > [26062.127174] 		ref#0: shared data backref parent 214908928 count 1
> > [26062.127176] 	item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66
> > [26062.127177] 		extent refs 2 gen 2381729 flags 1
> > [26062.127178] 		ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1
> > [26062.127180] 		ref#1: shared data backref parent 437808545792 count 1
> > [26062.127182] 	item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66
> > [26062.127183] 		extent refs 2 gen 2369098 flags 1
> > [26062.127184] 		ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1
> > [26062.127186] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.127188] 	item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66
> > [26062.127189] 		extent refs 2 gen 2368700 flags 1
> > [26062.127190] 		ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1
> > [26062.127191] 		ref#1: shared data backref parent 494105804800 count 1
> > [26062.127193] 	item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37
> > [26062.127195] 		extent refs 1 gen 1444854 flags 1
> > [26062.127195] 		ref#0: shared data backref parent 102547456 count 1
> > [26062.127197] 	item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79
> > [26062.127198] 		extent refs 3 gen 1800750 flags 1
> > [26062.127199] 		ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1
> > [26062.127201] 		ref#1: shared data backref parent 509783408640 count 1
> > [26062.127202] 		ref#2: shared data backref parent 508543451136 count 1
> > [26062.127204] 	item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37
> > [26062.127205] 		extent refs 1 gen 1444854 flags 1
> > [26062.127206] 		ref#0: shared data backref parent 93732864 count 1
> > [26062.127208] 	item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37
> > [26062.127209] 		extent refs 1 gen 1444854 flags 1
> > [26062.127210] 		ref#0: shared data backref parent 93732864 count 1
> > [26062.127213] 	item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53
> > [26062.127214] 		extent refs 1 gen 1444854 flags 1
> > [26062.127215] 		ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1
> > [26062.127217] 	item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92
> > [26062.127219] 		extent refs 4 gen 2419085 flags 1
> > [26062.127220] 		ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1
> > [26062.127222] 		ref#1: shared data backref parent 494075035648 count 1
> > [26062.127223] 		ref#2: shared data backref parent 470850240512 count 1
> > [26062.127225] 		ref#3: shared data backref parent 437034745856 count 1
> > [26062.127226] 	item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79
> > [26062.127228] 		extent refs 3 gen 1800750 flags 1
> > [26062.127229] 		ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1
> > [26062.127230] 		ref#1: shared data backref parent 509783408640 count 1
> > [26062.127232] 		ref#2: shared data backref parent 508543451136 count 1
> > [26062.127234] 	item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66
> > [26062.127235] 		extent refs 2 gen 2368731 flags 1
> > [26062.127236] 		ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1
> > [26062.127237] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.127239] 	item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37
> > [26062.127241] 		extent refs 1 gen 1444854 flags 1
> > [26062.127241] 		ref#0: shared data backref parent 93732864 count 1
> > [26062.127243] 	item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37
> > [26062.127244] 		extent refs 1 gen 1444854 flags 1
> > [26062.127245] 		ref#0: shared data backref parent 93732864 count 1
> > [26062.127247] 	item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37
> > [26062.127249] 		extent refs 1 gen 1444854 flags 1
> > [26062.127249] 		ref#0: shared data backref parent 93732864 count 1
> > [26062.127252] 	item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37
> > [26062.127253] 		extent refs 1 gen 1444854 flags 1
> > [26062.127254] 		ref#0: shared data backref parent 93732864 count 1
> > [26062.127257] 	item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37
> > [26062.127258] 		extent refs 1 gen 1444854 flags 1
> > [26062.127259] 		ref#0: shared data backref parent 93732864 count 1
> > [26062.127262] 	item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53
> > [26062.127263] 		extent refs 1 gen 2381741 flags 1
> > [26062.127264] 		ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1
> > [26062.127266] 	item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66
> > [26062.127268] 		extent refs 2 gen 2368882 flags 1
> > [26062.127269] 		ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1
> > [26062.127270] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.127272] 	item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66
> > [26062.127274] 		extent refs 2 gen 2368909 flags 1
> > [26062.127275] 		ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1
> > [26062.127277] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.127279] 	item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66
> > [26062.127280] 		extent refs 2 gen 2369146 flags 1
> > [26062.127281] 		ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1
> > [26062.127283] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.127285] 	item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37
> > [26062.127286] 		extent refs 1 gen 2369353 flags 1
> > [26062.127287] 		ref#0: shared data backref parent 510167498752 count 1
> > [26062.127289] 	item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79
> > [26062.127294] 		extent refs 3 gen 1800733 flags 1
> > [26062.127294] 		ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1
> > [26062.127296] 		ref#1: shared data backref parent 509783752704 count 1
> > [26062.127298] 		ref#2: shared data backref parent 509351313408 count 1
> > [26062.127299] 	item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183
> > [26062.127303] 		extent refs 22 gen 1800674 flags 1
> > [26062.127304] 		ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2
> > [26062.127306] 		ref#1: shared data backref parent 508152889344 count 2
> > [26062.127307] 		ref#2: shared data backref parent 508031139840 count 2
> > [26062.127308] 		ref#3: shared data backref parent 493563478016 count 2
> > [26062.127310] 		ref#4: shared data backref parent 493539983360 count 2
> > [26062.127311] 		ref#5: shared data backref parent 492674990080 count 2
> > [26062.127312] 		ref#6: shared data backref parent 475132166144 count 2
> > [26062.127314] 		ref#7: shared data backref parent 471032348672 count 2
> > [26062.127315] 		ref#8: shared data backref parent 436111769600 count 2
> > [26062.127317] 		ref#9: shared data backref parent 883359744 count 2
> > [26062.127318] 		ref#10: shared data backref parent 85082112 count 2
> > [26062.127320] 	item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183
> > [26062.127321] 		extent refs 33 gen 1800674 flags 1
> > [26062.127322] 		ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3
> > [26062.127324] 		ref#1: shared data backref parent 508152889344 count 3
> > [26062.127325] 		ref#2: shared data backref parent 508031139840 count 3
> > [26062.127327] 		ref#3: shared data backref parent 493563478016 count 3
> > [26062.127328] 		ref#4: shared data backref parent 493539983360 count 3
> > [26062.127329] 		ref#5: shared data backref parent 492674990080 count 3
> > [26062.127331] 		ref#6: shared data backref parent 475132166144 count 3
> > [26062.127333] 		ref#7: shared data backref parent 471032348672 count 3
> > [26062.127334] 		ref#8: shared data backref parent 436111769600 count 3
> > [26062.127335] 		ref#9: shared data backref parent 883359744 count 3
> > [26062.127337] 		ref#10: shared data backref parent 85082112 count 3
> > [26062.127338] 	item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92
> > [26062.127340] 		extent refs 4 gen 2419085 flags 1
> > [26062.127341] 		ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1
> > [26062.127342] 		ref#1: shared data backref parent 494075035648 count 1
> > [26062.127344] 		ref#2: shared data backref parent 470850240512 count 1
> > [26062.127345] 		ref#3: shared data backref parent 437034745856 count 1
> > [26062.127347] 	item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66
> > [26062.127348] 		extent refs 2 gen 2368932 flags 1
> > [26062.127349] 		ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1
> > [26062.127351] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.127353] 	item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37
> > [26062.127354] 		extent refs 1 gen 2369353 flags 1
> > [26062.127355] 		ref#0: shared data backref parent 510165155840 count 1
> > [26062.127356] 	item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66
> > [26062.127358] 		extent refs 2 gen 2369168 flags 1
> > [26062.127359] 		ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1
> > [26062.127360] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.127362] 	item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53
> > [26062.127364] 		extent refs 1 gen 2369305 flags 1
> > [26062.127365] 		ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1
> > [26062.127367] 	item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92
> > [26062.127369] 		extent refs 4 gen 2419085 flags 1
> > [26062.127370] 		ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1
> > [26062.127372] 		ref#1: shared data backref parent 494075035648 count 1
> > [26062.127374] 		ref#2: shared data backref parent 470850240512 count 1
> > [26062.127375] 		ref#3: shared data backref parent 437034745856 count 1
> > [26062.127377] 	item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66
> > [26062.127378] 		extent refs 2 gen 2368955 flags 1
> > [26062.127379] 		ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1
> > [26062.127382] 		ref#1: shared data backref parent 437615230976 count 1
> > [26062.127384] 	item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37
> > [26062.127385] 		extent refs 1 gen 2369352 flags 1
> > [26062.127386] 		ref#0: shared data backref parent 510178295808 count 1
> > [26062.127388] 	item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37
> > [26062.127390] 		extent refs 1 gen 2369352 flags 1
> > [26062.127391] 		ref#0: shared data backref parent 510058790912 count 1
> > [26062.127393] 	item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37
> > [26062.127394] 		extent refs 1 gen 2369352 flags 1
> > [26062.127395] 		ref#0: shared data backref parent 510059184128 count 1
> > [26062.127398] 	item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53
> > [26062.127399] 		extent refs 1 gen 2378552 flags 1
> > [26062.127400] 		ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1
> > [26062.127403] 	item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53
> > [26062.127405] 		extent refs 1 gen 2378552 flags 1
> > [26062.127406] 		ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1
> > [26062.127408] 	item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53
> > [26062.127409] 		extent refs 1 gen 2381741 flags 1
> > [26062.127410] 		ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1
> > [26062.127413] 	item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66
> > [26062.127414] 		extent refs 2 gen 2368700 flags 1
> > [26062.127415] 		ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1
> > [26062.127416] 		ref#1: shared data backref parent 494105804800 count 1
> > [26062.127419] 	item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66
> > [26062.127420] 		extent refs 2 gen 2368700 flags 1
> > [26062.127421] 		ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1
> > [26062.127423] 		ref#1: shared data backref parent 494105804800 count 1
> > [26062.127425] 	item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66
> > [26062.127426] 		extent refs 2 gen 2368700 flags 1
> > [26062.127428] 		ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1
> > [26062.127430] 		ref#1: shared data backref parent 494105804800 count 1
> > [26062.127432] 	item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183
> > [26062.127434] 		extent refs 11 gen 1800699 flags 1
> > [26062.127435] 		ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1
> > [26062.127437] 		ref#1: shared data backref parent 508152987648 count 1
> > [26062.127439] 		ref#2: shared data backref parent 508031172608 count 1
> > [26062.127440] 		ref#3: shared data backref parent 493563822080 count 1
> > [26062.127441] 		ref#4: shared data backref parent 493540130816 count 1
> > [26062.127443] 		ref#5: shared data backref parent 492675170304 count 1
> > [26062.127444] 		ref#6: shared data backref parent 475884584960 count 1
> > [26062.127446] 		ref#7: shared data backref parent 471032496128 count 1
> > [26062.127447] 		ref#8: shared data backref parent 436111884288 count 1
> > [26062.127448] 		ref#9: shared data backref parent 883703808 count 1
> > [26062.127450] 		ref#10: shared data backref parent 85671936 count 1
> > [26062.127452] 	item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37
> > [26062.127453] 		extent refs 1 gen 1444854 flags 1
> > [26062.127454] 		ref#0: shared data backref parent 214908928 count 1
> > [26062.127457] 	item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37
> > [26062.127458] 		extent refs 1 gen 2426255 flags 1
> > [26062.127459] 		ref#0: shared data backref parent 436271005696 count 1
> > [26062.127461] 	item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183
> > [26062.127463] 		extent refs 22 gen 1800674 flags 1
> > [26062.127464] 		ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2
> > [26062.127465] 		ref#1: shared data backref parent 508152987648 count 2
> > [26062.127467] 		ref#2: shared data backref parent 508031172608 count 2
> > [26062.127468] 		ref#3: shared data backref parent 493563822080 count 2
> > [26062.127470] 		ref#4: shared data backref parent 493540130816 count 2
> > [26062.127472] 		ref#5: shared data backref parent 492675170304 count 2
> > [26062.127473] 		ref#6: shared data backref parent 475884584960 count 2
> > [26062.127475] 		ref#7: shared data backref parent 471032496128 count 2
> > [26062.127476] 		ref#8: shared data backref parent 436111884288 count 2
> > [26062.127478] 		ref#9: shared data backref parent 883703808 count 2
> > [26062.127480] 		ref#10: shared data backref parent 85671936 count 2
> > [26062.127483] 	item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37
> > [26062.127486] 		extent refs 1 gen 1444854 flags 1
> > [26062.127488] 		ref#0: shared data backref parent 167198720 count 1
> > [26062.127491] 	item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66
> > [26062.127494] 		extent refs 2 gen 1400856 flags 1
> > [26062.127496] 		ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1
> > [26062.127498] 		ref#1: shared data backref parent 470954196992 count 1
> > [26062.127501] 	item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37
> > [26062.127503] 		extent refs 1 gen 1444854 flags 1
> > [26062.127504] 		ref#0: shared data backref parent 93732864 count 1
> > [26062.127507] 	item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53
> > [26062.127510] 		extent refs 1 gen 2381741 flags 1
> > [26062.127512] 		ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1
> > [26062.127518] 	item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66
> > [26062.127520] 		extent refs 2 gen 2368701 flags 1
> > [26062.127521] 		ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1
> > [26062.127523] 		ref#1: shared data backref parent 494107131904 count 1
> > [26062.127526] 	item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183
> > [26062.127527] 		extent refs 44 gen 1800674 flags 1
> > [26062.127528] 		ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4
> > [26062.127530] 		ref#1: shared data backref parent 508152987648 count 4
> > [26062.127532] 		ref#2: shared data backref parent 508031172608 count 4
> > [26062.127533] 		ref#3: shared data backref parent 493563822080 count 4
> > [26062.127535] 		ref#4: shared data backref parent 493540130816 count 4
> > [26062.127537] 		ref#5: shared data backref parent 492675170304 count 4
> > [26062.127539] 		ref#6: shared data backref parent 475884584960 count 4
> > [26062.127540] 		ref#7: shared data backref parent 471032496128 count 4
> > [26062.127542] 		ref#8: shared data backref parent 436111884288 count 4
> > [26062.127543] 		ref#9: shared data backref parent 883703808 count 4
> > [26062.127545] 		ref#10: shared data backref parent 85671936 count 4
> > [26062.127547] 	item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79
> > [26062.127549] 		extent refs 3 gen 1800750 flags 1
> > [26062.127552] 		ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1
> > [26062.127554] 		ref#1: shared data backref parent 509783130112 count 1
> > [26062.127557] 		ref#2: shared data backref parent 508544876544 count 1
> > [26062.127560] 	item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37
> > [26062.127561] 		extent refs 1 gen 1444854 flags 1
> > [26062.127562] 		ref#0: shared data backref parent 93732864 count 1
> > [26062.127565] 	item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37
> > [26062.127566] 		extent refs 1 gen 1444854 flags 1
> > [26062.127568] 		ref#0: shared data backref parent 214908928 count 1
> > [26062.127570] 	item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37
> > [26062.127572] 		extent refs 1 gen 1444854 flags 1
> > [26062.127573] 		ref#0: shared data backref parent 93732864 count 1
> > [26062.127577] 	item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24
> > [26062.127580] 		block group used 834060288 chunk_objectid 256 flags 1
> > [26062.127581] 	item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37
> > [26062.127583] 		extent refs 1 gen 1444854 flags 1
> > [26062.127584] 		ref#0: shared data backref parent 167198720 count 1
> > [26062.127586] 	item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37
> > [26062.127587] 		extent refs 1 gen 1444854 flags 1
> > [26062.127588] 		ref#0: shared data backref parent 167198720 count 1
> > [26062.127590] 	item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37
> > [26062.127591] 		extent refs 1 gen 1444854 flags 1
> > [26062.127592] 		ref#0: shared data backref parent 167198720 count 1
> > [26062.127594] 	item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37
> > [26062.127595] 		extent refs 1 gen 1444854 flags 1
> > [26062.127596] 		ref#0: shared data backref parent 167198720 count 1
> > [26062.127598] 	item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37
> > [26062.127599] 		extent refs 1 gen 1444854 flags 1
> > [26062.127600] 		ref#0: shared data backref parent 167198720 count 1
> > [26062.127602] 	item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37
> > [26062.127603] 		extent refs 1 gen 1444854 flags 1
> > [26062.127604] 		ref#0: shared data backref parent 167198720 count 1
> > [26062.127606] 	item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183
> > [26062.127607] 		extent refs 11 gen 2409230 flags 1
> > [26062.127608] 		ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1
> > [26062.127610] 		ref#1: shared data backref parent 508153675776 count 1
> > [26062.127611] 		ref#2: shared data backref parent 493566640128 count 1
> > [26062.127613] 		ref#3: shared data backref parent 493540999168 count 1
> > [26062.127614] 		ref#4: shared data backref parent 492676497408 count 1
> > [26062.127616] 		ref#5: shared data backref parent 475884634112 count 1
> > [26062.127617] 		ref#6: shared data backref parent 475691106304 count 1
> > [26062.127619] 		ref#7: shared data backref parent 471034494976 count 1
> > [26062.127620] 		ref#8: shared data backref parent 436113883136 count 1
> > [26062.127622] 		ref#9: shared data backref parent 886407168 count 1
> > [26062.127623] 		ref#10: shared data backref parent 89440256 count 1
> > [26062.127625] 	item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170
> > [26062.127626] 		extent refs 10 gen 2413323 flags 1
> > [26062.127629] 		ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1
> > [26062.127631] 		ref#1: shared data backref parent 508153675776 count 1
> > [26062.127633] 		ref#2: shared data backref parent 493566640128 count 1
> > [26062.127635] 		ref#3: shared data backref parent 493540999168 count 1
> > [26062.127636] 		ref#4: shared data backref parent 492676497408 count 1
> > [26062.127638] 		ref#5: shared data backref parent 475691106304 count 1
> > [26062.127640] 		ref#6: shared data backref parent 471034494976 count 1
> > [26062.127642] 		ref#7: shared data backref parent 436113883136 count 1
> > [26062.127643] 		ref#8: shared data backref parent 886407168 count 1
> > [26062.127645] 		ref#9: shared data backref parent 89440256 count 1
> > [26062.127647] 	item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37
> > [26062.127648] 		extent refs 1 gen 2407793 flags 1
> > [26062.127649] 		ref#0: shared data backref parent 475884634112 count 1
> > [26062.127653] 	item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183
> > [26062.127655] 		extent refs 11 gen 1800729 flags 1
> > [26062.127656] 		ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1
> > [26062.127658] 		ref#1: shared data backref parent 508153069568 count 1
> > [26062.127659] 		ref#2: shared data backref parent 508031270912 count 1
> > [26062.127661] 		ref#3: shared data backref parent 493564215296 count 1
> > [26062.127663] 		ref#4: shared data backref parent 493540098048 count 1
> > [26062.127664] 		ref#5: shared data backref parent 492676038656 count 1
> > [26062.127666] 		ref#6: shared data backref parent 475884142592 count 1
> > [26062.127668] 		ref#7: shared data backref parent 471032840192 count 1
> > [26062.127669] 		ref#8: shared data backref parent 436112146432 count 1
> > [26062.127671] 		ref#9: shared data backref parent 883752960 count 1
> > [26062.127674] 		ref#10: shared data backref parent 86654976 count 1
> > [26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456  owner 73020573 offset 17039360
> > [26062.127680] ------------[ cut here ]------------
> > [26062.127682] BTRFS: Transaction aborted (error -2)
> > [26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921
> > [26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd
 _hda_intel
> > [26062.127733]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
> > [26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
> > [26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
> > [26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921
> > [26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df
> > [26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282
> > [26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007
> > [26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550
> > [26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
> > [26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169
> > [26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
> > [26062.127815] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
> > [26062.127817] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
> > [26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > [26062.127823] Call Trace:
> > [26062.127832]  __btrfs_run_delayed_refs+0x750/0xc36
> > [26062.127837]  ? __switch_to_asm+0x41/0x70
> > [26062.127839]  ? __switch_to_asm+0x35/0x70
> > [26062.127841]  ? __switch_to_asm+0x41/0x70
> > [26062.127848]  ? __switch_to+0x13d/0x3d5
> > [26062.127852]  btrfs_run_delayed_refs+0x5d/0x132
> > [26062.127855]  btrfs_commit_transaction+0x55/0x7c8
> > [26062.127858]  ? start_transaction+0x347/0x3cb
> > [26062.127862]  transaction_kthread+0xc9/0x135
> > [26062.127865]  ? btrfs_cleanup_transaction+0x403/0x403
> > [26062.127869]  kthread+0xeb/0xf0
> > [26062.127872]  ? kthread_create_worker_on_cpu+0x65/0x65
> > [26062.127875]  ret_from_fork+0x35/0x40
> > [26062.127879] ---[ end trace 4c1a6b3749a2f651 ]---
> > [26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry
> > [26062.127917] BTRFS info (device dm-2): forced readonly
> > [26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entry
> 
> On Tue, Jul 17, 2018 at 05:24:51PM -0700, Marc MERLIN wrote:
> > On Wed, Jul 18, 2018 at 08:05:51AM +0800, Qu Wenruo wrote:
> > > No OOM triggers? That's a little strange.
> > > Maybe it's related to how kernel handles memory over-commit?
> >  
> > Yes, I think you are correct.
> > 
> > > And for the hang, I think it's related to some memory allocation failure
> > > and error handler just didn't handle it well, so it's causing deadlock
> > > for certain page.
> > 
> > That indeed matches what I'm seeing.
> > 
> > > ENOMEM handling is pretty common but hardly verified, so it's not that
> > > strange, but we must locate the problem.
> > 
> > I seem to be getting deadlocks in the kernel, so I'm hoping that at least
> > it's checked there, but maybe not?
> > 
> > > In my system, at least I'm not using btrfs as root fs, and for the
> > > memory eating program I normally ensure it's eating all the memory +
> > > swap, so OOM killer is always triggered, maybe that's the cause.
> > > 
> > > So in your case, maybe it's btrfs not really taking up all memory, thus
> > > OOM killer not triggered.
> > 
> > Correct, the swap is not used.
> > 
> > > Any kernel dmesg about OOM killer triggered?
> >  
> > Nothing at all. It never gets triggered.
> > 
> > > > Here is my system when it virtually died:
> > > > ER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> > > > root     31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49   1:35 ./btrfs check /dev/mapper/dshelf2
> > 
> > See how btrs was taking 29GB in that ps output (that's before it takes
> > everything and I can't even type ps anymore)
> > Note that VSZ is almost equal to RSS. Nothing gets swapped.
> > 
> > Then see free output:
> > 
> > > >              total       used       free     shared    buffers     cached
> > > > Mem:      32643788   32180100     463688          0      44664     119508
> > > > -/+ buffers/cache:   32015928     627860
> > > > Swap:     15616764     443676   15173088
> > > 
> > > For swap, it looks like only some other program's memory is swapped out,
> > > not btrfs'.
> > 
> > That's exactly correct. btrfs check never goes to swap, I'm not sure why,
> > and because there is virtual memory free, maybe that's why OOM does not
> > trigger?
> > So I guess I can probably "fix" my problem by removing swap, but ultimately
> > it would be useful to know why memory taken by btrfs check does not end up
> > in swap.
> > 
> > > And unfortunately, I'm not so familiar with OOM/MM code outside of
> > > filesystem.
> > > Any help from other experienced developers would definitely help to
> > > solve why memory of 'btrfs check' is not swapped out or why OOM killer
> > > is not triggered.
> > 
> > Do you have someone from linux-vm you might be able to ask, or should we Cc
> > this thread there?
> > 
> > Thanks,
> > Marc
> > -- 
> > "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> > Microsoft is to operating systems ....
> >                                       .... what McDonalds is to gourmet cooking
> > Home page: http://marc.merlins.org/  
> 
> On Tue, Jul 17, 2018 at 01:59:05PM -0700, Marc MERLIN wrote:
> > Ok, I did more testing. Qu is right that btrfs check does not crash the kernel.
> > It just takes all the memory until linux hangs everywhere, and somehow (no idea why) 
> > the OOM killer never triggers.
> > Details below:
> > 
> > On Tue, Jul 17, 2018 at 01:32:57PM -0700, Marc MERLIN wrote:
> > > Here is what I got when the system was not doing well (it took minutes to run):
> > > 
> > >              total       used       free     shared    buffers     cached
> > > Mem:      32643788   32070952     572836          0     102160    4378772
> > > -/+ buffers/cache:   27590020    5053768
> > > Swap:     15616764     973596   14643168
> > 
> > ok, the reason it was not that close to 0 was due to /dev/shm it seems.
> > I cleared that, and now I can get it to go to near 0 again.
> > I'm wrong about the system being fully crashed, it's not, it's just very
> > close to being hung.
> > I can type killall -9 btrfs in the serial console and wait a few minutes.
> > The system eventually recovers, but it's impossible to fix anything via ssh 
> > apparently because networking does not get to run when I'm in this state.
> > 
> > I'm not sure why my system reproduces this easy while Qu's system does not, 
> > but Qu was right that the kernel is not dead and that it's merely a problem of userspace
> > taking all the RAM and somehow not being killed by OOM
> > 
> > I checked the PID and don't see why it's not being killed:
> > gargamel:/proc/31006# grep . oom*
> > oom_adj:0
> > oom_score:221   << this increases a lot, but OOM never kills it
> > oom_score_adj:0
> > 
> > I have these variables:
> > /proc/sys/vm/oom_dump_tasks:1
> > /proc/sys/vm/oom_kill_allocating_task:0
> > /proc/sys/vm/overcommit_kbytes:0
> > /proc/sys/vm/overcommit_memory:0
> > /proc/sys/vm/overcommit_ratio:50  << is this bad (seems default)
> > 
> > Here is my system when it virtually died:
> > ER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> > root     31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49   1:35 ./btrfs check /dev/mapper/dshelf2
> > 
> >              total       used       free     shared    buffers     cached
> > Mem:      32643788   32180100     463688          0      44664     119508
> > -/+ buffers/cache:   32015928     627860
> > Swap:     15616764     443676   15173088
> > 
> > MemTotal:       32643788 kB
> > MemFree:          463440 kB
> > MemAvailable:      44864 kB
> > Buffers:           44664 kB
> > Cached:           120360 kB
> > SwapCached:        87064 kB
> > Active:         30381404 kB
> > Inactive:         585952 kB
> > Active(anon):   30334696 kB
> > Inactive(anon):   474624 kB
> > Active(file):      46708 kB
> > Inactive(file):   111328 kB
> > Unevictable:        5616 kB
> > Mlocked:            5616 kB
> > SwapTotal:      15616764 kB
> > SwapFree:       15173088 kB
> > Dirty:              1636 kB
> > Writeback:             4 kB
> > AnonPages:      30734240 kB
> > Mapped:            67236 kB
> > Shmem:              3036 kB
> > Slab:             267884 kB
> > SReclaimable:      51528 kB
> > SUnreclaim:       216356 kB
> > KernelStack:       10144 kB
> > PageTables:        69284 kB
> > NFS_Unstable:          0 kB
> > Bounce:                0 kB
> > WritebackTmp:          0 kB
> > CommitLimit:    31938656 kB
> > Committed_AS:   32865492 kB
> > VmallocTotal:   34359738367 kB
> > VmallocUsed:           0 kB
> > VmallocChunk:          0 kB
> > HardwareCorrupted:     0 kB
> > AnonHugePages:         0 kB
> > ShmemHugePages:        0 kB
> > ShmemPmdMapped:        0 kB
> > CmaTotal:          16384 kB
> > CmaFree:               0 kB
> > HugePages_Total:       0
> > HugePages_Free:        0
> > HugePages_Rsvd:        0
> > HugePages_Surp:        0
> > Hugepagesize:       2048 kB
> > Hugetlb:               0 kB
> > DirectMap4k:      560404 kB
> > DirectMap2M:    32692224 kB
> > 
> > 
> > -- 
> > "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> > Microsoft is to operating systems ....
> >                                       .... what McDonalds is to gourmet cooking
> > Home page: http://marc.merlins.org/  
> 
> 
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/  
> 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
  2019-10-23  0:55           ` Marc MERLIN
@ 2019-10-26  3:36             ` Marc MERLIN
  0 siblings, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2019-10-26  3:36 UTC (permalink / raw)
  To: Linux BTRFS Mailinglist, Qu Wenruo, Su Yue

So, no reply at all.
No one cares about massive filesystem corruption in a recent enough
kernel?

On Tue, Oct 22, 2019 at 05:55:53PM -0700, Marc MERLIN wrote:
> On Fri, Oct 18, 2019 at 08:07:28PM -0700, Marc MERLIN wrote:
> > Ok, so before blowing the filesystem away after it was apparently badly
> > damaged by a suspend to disk, I tried check --repair and I hit an
> > infinite loop.
> > 
> > Let me know if you'd like anything off the FS before I delete it.
> 
> I heard nothing back, so I deleted the FS and restored from backup.
> 
> But now I'm scared of ever doing a suspend to disk again.
> Could someone please look at the logs and give me some idea of what
> happened, if at all possible?
> 
> Non recoverable data corruption on my laptop when I travel and
> backups/restores are complicated, is a bit unnerving...
> 
> Thanks,
> Marc
> 
> > Thanks,
> > Marc
> > 
> > enabling repair mode
> > repair mode will force to clear out log tree, are you sure? [y/N]: y
> > Checking filesystem on /dev/mapper/pool1
> > UUID: fda628bc-1ca4-49c5-91c2-4260fe967a23
> > checking extents
> > Backref 415334400 parent 36028797198598144 not referenced back 0x5648ef1870e0
> > Backref 415334400 parent 179634176 root 179634176 not found in extent tree
> > Incorrect global backref count on 415334400 found 2 wanted 1
> > backpointer mismatch on [415334400 16384]
> > repair deleting extent record: key 415334400 169 0
> > adding new tree backref on start 415334400 len 16384 parent 179634176 root 179634176
> > Repaired extent references for 415334400
> > ref mismatch on [101995261952 4096] extent item 36028797018963969, found 1
> > repair deleting extent record: key 101995261952 168 4096
> > adding new data backref on 101995261952 root 456 owner 74455677 offset 64892928 found 1
> > Repaired extent references for 101995261952
> > Incorrect local backref count on 458640384000 root 456 owner 81409181 offset 17039360 found 0 wanted 1 back 0x5648eefd3d10
> > Backref disk bytenr does not match extent record, bytenr=458640384000, ref bytenr=0
> > Backref 458640384000 root 456 owner 73020573 offset 17039360 num_refs 0 not found in extent tree
> > Incorrect local backref count on 458640384000 root 456 owner 73020573 offset 17039360 found 1 wanted 0 back 0x5648b32a9600
> > backpointer mismatch on [458640384000 86016]
> > repair deleting extent record: key 458640384000 168 86016
> > adding new data backref on 458640384000 parent 438017720320 owner 0 offset 0 found 1
> > adding new data backref on 458640384000 root 456 owner 73020573 offset 17039360 found 1
> > Repaired extent references for 458640384000
> > Fixed 0 roots.
> > checking free space cache
> > cache and super generation don't match, space cache will be invalidated
> > checking fs roots
> > Deleting bad dir index [10138517,96,436945] root 456
> > Deleting bad dir index [10138518,96,646273] root 456
> > Deleting bad dir index [10138517,96,437016] root 456
> > Deleting bad dir index [10138518,96,808999] root 456
> > Deleting bad dir index [10215134,96,149427] root 456
> > Deleting bad dir index [10240541,96,268037] root 456
> > Deleting bad dir index [10138517,96,540247] root 456
> > Deleting bad dir index [10138518,96,825234] root 456
> > Deleting bad dir index [10138517,96,736673] root 456
> > Deleting bad dir index [10138518,96,1118221] root 456
> > Deleting bad dir index [10240541,96,439703] root 456
> > Deleting bad dir index [10138517,96,752282] root 456
> > root 456 inode 75431563 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 4096, len: 4096
> > root 456 inode 75431568 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 1638400
> > root 456 inode 75431583 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 147456
> > root 456 inode 75431585 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 208896
> > root 456 inode 75431591 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 2523136
> > root 456 inode 75431730 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 208896
> > root 456 inode 75431744 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 2084864
> > root 456 inode 75431751 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 172032
> > root 456 inode 75431756 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 8192
> > root 456 inode 75431760 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 12288
> > root 456 inode 75431765 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 32768
> > root 456 inode 75431773 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 90112
> > Fixed discount file extents for inode: 75432421 in root: 456
> > Fixed discount file extents for inode: 75432429 in root: 456
> > root 456 inode 75432429 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 4096
> > Fixed discount file extents for inode: 75432430 in root: 456
> > Fixed discount file extents for inode: 75432432 in root: 456
> > Fixed discount file extents for inode: 75432433 in root: 456
> > root 456 inode 75432433 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 4096
> > root 456 inode 75432434 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 724992
> > Fixed discount file extents for inode: 75432436 in root: 456
> > Fixed discount file extents for inode: 75432437 in root: 456
> > root 456 inode 75432437 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 4096
> > Fixed discount file extents for inode: 75432438 in root: 456
> > Fixed discount file extents for inode: 75432441 in root: 456
> > root 456 inode 75432441 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 4096
> > Fixed discount file extents for inode: 75432444 in root: 456
> > root 456 inode 75432444 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 4096
> > Fixed discount file extents for inode: 75432445 in root: 456
> > Fixed discount file extents for inode: 75432447 in root: 456
> > Fixed discount file extents for inode: 75432448 in root: 456
> > Fixed discount file extents for inode: 75432454 in root: 456
> > Fixed discount file extents for inode: 75432457 in root: 456
> > root 456 inode 75432457 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 4096
> > root 456 inode 75432471 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 20480
> > Fixed discount file extents for inode: 75432473 in root: 456
> > root 456 inode 75432473 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 20480
> > Fixed discount file extents for inode: 75432482 in root: 456
> > Fixed discount file extents for inode: 75432483 in root: 456
> > Fixed discount file extents for inode: 75432484 in root: 456
> > Fixed discount file extents for inode: 75432487 in root: 456
> > Fixed discount file extents for inode: 75432489 in root: 456
> > Fixed discount file extents for inode: 75432494 in root: 456
> > Fixed discount file extents for inode: 75432611 in root: 456
> > Fixed discount file extents for inode: 75432622 in root: 456
> > Fixed discount file extents for inode: 75432626 in root: 456
> > Fixed discount file extents for inode: 75432628 in root: 456
> > Fixed discount file extents for inode: 75432636 in root: 456
> > Fixed discount file extents for inode: 75432637 in root: 456
> > root 456 inode 75432637 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 4096
> > Fixed discount file extents for inode: 75432643 in root: 456
> > root 456 inode 75432643 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 135168
> > Fixed discount file extents for inode: 75432650 in root: 456
> > root 456 inode 75432650 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 4096
> > Fixed discount file extents for inode: 75432656 in root: 456
> > root 456 inode 75432656 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 4096
> > Fixed discount file extents for inode: 75432657 in root: 456
> > root 456 inode 75432657 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 4096
> > Fixed discount file extents for inode: 75432659 in root: 456
> > Fixed discount file extents for inode: 75432662 in root: 456
> > root 456 inode 75432662 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 8192
> > Fixed discount file extents for inode: 75432663 in root: 456
> > root 456 inode 75432663 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 8192
> > Fixed discount file extents for inode: 75432664 in root: 456
> > root 456 inode 75432664 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 8192
> > Fixed discount file extents for inode: 75432674 in root: 456
> > root 456 inode 75432674 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 8192
> > Fixed discount file extents for inode: 75432692 in root: 456
> > root 456 inode 75432692 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 4096
> > Fixed discount file extents for inode: 75432708 in root: 456
> > Fixed discount file extents for inode: 75432709 in root: 456
> > root 456 inode 75432709 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 4096
> > Fixed discount file extents for inode: 75432712 in root: 456
> > Fixed discount file extents for inode: 75432717 in root: 456
> > Fixed discount file extents for inode: 75432720 in root: 456
> > Fixed discount file extents for inode: 75432725 in root: 456
> > Fixed discount file extents for inode: 75432731 in root: 456
> > Fixed discount file extents for inode: 75432732 in root: 456
> > Fixed discount file extents for inode: 75432749 in root: 456
> > Fixed discount file extents for inode: 75432762 in root: 456
> > Fixed discount file extents for inode: 75432765 in root: 456
> > Fixed discount file extents for inode: 75432783 in root: 456
> > Fixed discount file extents for inode: 75432784 in root: 456
> > Fixed discount file extents for inode: 75432785 in root: 456
> > root 456 inode 75432785 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 8192
> > Fixed discount file extents for inode: 75432786 in root: 456
> > root 456 inode 75432786 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 8192
> > Fixed discount file extents for inode: 75432796 in root: 456
> > root 456 inode 75432796 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 8192
> > Fixed discount file extents for inode: 75432799 in root: 456
> > root 456 inode 75432799 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 8192
> > Fixed discount file extents for inode: 75432801 in root: 456
> > root 456 inode 75432801 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 4096
> > Fixed discount file extents for inode: 75432807 in root: 456
> > Fixed discount file extents for inode: 75432817 in root: 456
> > Fixed discount file extents for inode: 75432829 in root: 456
> > Fixed discount file extents for inode: 75432860 in root: 456
> > root 456 inode 75432860 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 4096
> > Fixed discount file extents for inode: 75432862 in root: 456
> > Fixed discount file extents for inode: 75432863 in root: 456
> > Fixed discount file extents for inode: 75432869 in root: 456
> > Fixed discount file extents for inode: 75432870 in root: 456
> > Fixed discount file extents for inode: 75432871 in root: 456
> > Fixed discount file extents for inode: 75432872 in root: 456
> > Fixed discount file extents for inode: 75432875 in root: 456
> > Fixed discount file extents for inode: 75432877 in root: 456
> > Fixed discount file extents for inode: 75432882 in root: 456
> > Fixed discount file extents for inode: 75432883 in root: 456
> > Fixed discount file extents for inode: 75432893 in root: 456
> > Fixed discount file extents for inode: 75432894 in root: 456
> > Fixed discount file extents for inode: 75432897 in root: 456
> > Fixed discount file extents for inode: 75432899 in root: 456
> > Fixed discount file extents for inode: 75432900 in root: 456
> > Fixed discount file extents for inode: 75432905 in root: 456
> > Fixed discount file extents for inode: 75432906 in root: 456
> > Fixed discount file extents for inode: 75432916 in root: 456
> > Fixed discount file extents for inode: 75432917 in root: 456
> > Fixed discount file extents for inode: 75432919 in root: 456
> > Fixed discount file extents for inode: 75432920 in root: 456
> > Fixed discount file extents for inode: 75432923 in root: 456
> > Fixed discount file extents for inode: 75432942 in root: 456
> > Fixed discount file extents for inode: 75432944 in root: 456
> > Fixed discount file extents for inode: 75432948 in root: 456
> > root 456 inode 75432948 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 8192
> > Fixed discount file extents for inode: 75432949 in root: 456
> > root 456 inode 75432949 errors 100, file extent discount
> > Found file extent holes:
> > 	start: 0, len: 8192
> > and it loops forever on 456
> > 
> > On Thu, Oct 17, 2019 at 07:56:04PM -0700, Marc MERLIN wrote:
> > > This happened almost after a resume from suspend to disk.
> > > First corruption and read only I got a very long time.
> > > 
> > > Could they be related?
> > > 
> > > [26062.126505] ------------[ cut here ]------------
> > > [26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
> > > [26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc s
 nd_hda_intel
> > > [26062.126561]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
> > > [26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
> > > [26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
> > > [26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921
> > > [26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41
> > > [26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246
> > > [26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000
> > > [26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558
> > > [26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
> > > [26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169
> > > [26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
> > > [26062.126625] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
> > > [26062.126627] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
> > > [26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > [26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > > [26062.126636] Call Trace:
> > > [26062.126647]  __btrfs_run_delayed_refs+0x750/0xc36
> > > [26062.126653]  ? __switch_to_asm+0x41/0x70
> > > [26062.126655]  ? __switch_to_asm+0x35/0x70
> > > [26062.126658]  ? __switch_to_asm+0x41/0x70
> > > [26062.126662]  ? __switch_to+0x13d/0x3d5
> > > [26062.126668]  btrfs_run_delayed_refs+0x5d/0x132
> > > [26062.126672]  btrfs_commit_transaction+0x55/0x7c8
> > > [26062.126676]  ? start_transaction+0x347/0x3cb
> > > [26062.126679]  transaction_kthread+0xc9/0x135
> > > [26062.126683]  ? btrfs_cleanup_transaction+0x403/0x403
> > > [26062.126688]  kthread+0xeb/0xf0
> > > [26062.126692]  ? kthread_create_worker_on_cpu+0x65/0x65
> > > [26062.126695]  ret_from_fork+0x35/0x40
> > > [26062.126698] ---[ end trace 4c1a6b3749a2f650 ]---
> > > [26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2
> > > [26062.126706] 	item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66
> > > [26062.126708] 		extent refs 2 gen 2369265 flags 1
> > > [26062.126709] 		ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1
> > > [26062.126711] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.126714] 	item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66
> > > [26062.126715] 		extent refs 2 gen 2369025 flags 1
> > > [26062.126716] 		ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1
> > > [26062.126718] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.126720] 	item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183
> > > [26062.126722] 		extent refs 11 gen 1800715 flags 1
> > > [26062.126722] 		ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1
> > > [26062.126724] 		ref#1: shared data backref parent 508153839616 count 1
> > > [26062.126726] 		ref#2: shared data backref parent 493541244928 count 1
> > > [26062.126727] 		ref#3: shared data backref parent 492677332992 count 1
> > > [26062.126728] 		ref#4: shared data backref parent 492328566784 count 1
> > > [26062.126730] 		ref#5: shared data backref parent 492245188608 count 1
> > > [26062.126731] 		ref#6: shared data backref parent 475885109248 count 1
> > > [26062.126732] 		ref#7: shared data backref parent 471036608512 count 1
> > > [26062.126733] 		ref#8: shared data backref parent 436114243584 count 1
> > > [26062.126735] 		ref#9: shared data backref parent 886718464 count 1
> > > [26062.126736] 		ref#10: shared data backref parent 90259456 count 1
> > > [26062.126738] 	item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66
> > > [26062.126740] 		extent refs 2 gen 2420471 flags 1
> > > [26062.126741] 		ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1
> > > [26062.126743] 		ref#1: shared data backref parent 41020112896 count 1
> > > [26062.126745] 	item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53
> > > [26062.126746] 		extent refs 1 gen 2422941 flags 1
> > > [26062.126747] 		ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1
> > > [26062.126749] 	item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53
> > > [26062.126750] 		extent refs 1 gen 1444854 flags 1
> > > [26062.126751] 		ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1
> > > [26062.126753] 	item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37
> > > [26062.126754] 		extent refs 1 gen 1444854 flags 1
> > > [26062.126755] 		ref#0: shared data backref parent 214892544 count 1
> > > [26062.126757] 	item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37
> > > [26062.126758] 		extent refs 1 gen 1444854 flags 1
> > > [26062.126759] 		ref#0: shared data backref parent 102547456 count 1
> > > [26062.126761] 	item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66
> > > [26062.126762] 		extent refs 2 gen 2422644 flags 1
> > > [26062.126763] 		ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1
> > > [26062.126765] 		ref#1: shared data backref parent 470658482176 count 1
> > > [26062.126768] 	item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66
> > > [26062.126769] 		extent refs 2 gen 2369217 flags 1
> > > [26062.126770] 		ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1
> > > [26062.126772] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.126774] 	item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66
> > > [26062.126775] 		extent refs 2 gen 2369242 flags 1
> > > [26062.126776] 		ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1
> > > [26062.126778] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.126780] 	item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53
> > > [26062.126781] 		extent refs 1 gen 2369305 flags 1
> > > [26062.126782] 		ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1
> > > [26062.126784] 	item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66
> > > [26062.126786] 		extent refs 2 gen 2369052 flags 1
> > > [26062.126787] 		ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1
> > > [26062.126788] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.126790] 	item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53
> > > [26062.126792] 		extent refs 1 gen 2369305 flags 1
> > > [26062.126793] 		ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1
> > > [26062.126795] 	item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66
> > > [26062.126796] 		extent refs 2 gen 2368754 flags 1
> > > [26062.126797] 		ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1
> > > [26062.126799] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.126801] 	item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183
> > > [26062.126802] 		extent refs 11 gen 1800682 flags 1
> > > [26062.126803] 		ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1
> > > [26062.126805] 		ref#1: shared data backref parent 508153069568 count 1
> > > [26062.126806] 		ref#2: shared data backref parent 508031270912 count 1
> > > [26062.126808] 		ref#3: shared data backref parent 493564215296 count 1
> > > [26062.126809] 		ref#4: shared data backref parent 493540098048 count 1
> > > [26062.126811] 		ref#5: shared data backref parent 492676038656 count 1
> > > [26062.126813] 		ref#6: shared data backref parent 475884142592 count 1
> > > [26062.126814] 		ref#7: shared data backref parent 471032840192 count 1
> > > [26062.126816] 		ref#8: shared data backref parent 436112146432 count 1
> > > [26062.126817] 		ref#9: shared data backref parent 883752960 count 1
> > > [26062.126818] 		ref#10: shared data backref parent 86654976 count 1
> > > [26062.126820] 	item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50
> > > [26062.126822] 		extent refs 2 gen 2369347 flags 1
> > > [26062.126823] 		ref#0: shared data backref parent 492981370880 count 1
> > > [26062.126824] 		ref#1: shared data backref parent 438017720320 count 1
> > > [26062.126826] 	item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50
> > > [26062.126827] 		extent refs 2 gen 2369347 flags 1
> > > [26062.126828] 		ref#0: shared data backref parent 492981370880 count 1
> > > [26062.126830] 		ref#1: shared data backref parent 438017720320 count 1
> > > [26062.126832] 	item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50
> > > [26062.126833] 		extent refs 2 gen 2369347 flags 1
> > > [26062.126834] 		ref#0: shared data backref parent 492981370880 count 1
> > > [26062.126836] 		ref#1: shared data backref parent 438017720320 count 1
> > > [26062.126839] 	item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37
> > > [26062.126840] 		extent refs 1 gen 2369352 flags 1
> > > [26062.126841] 		ref#0: shared data backref parent 510058790912 count 1
> > > [26062.126843] 	item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37
> > > [26062.126845] 		extent refs 1 gen 2369352 flags 1
> > > [26062.126845] 		ref#0: shared data backref parent 510058790912 count 1
> > > [26062.126848] 	item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37
> > > [26062.126849] 		extent refs 1 gen 2369352 flags 1
> > > [26062.126850] 		ref#0: shared data backref parent 510058790912 count 1
> > > [26062.126852] 	item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50
> > > [26062.126853] 		extent refs 2 gen 2369347 flags 1
> > > [26062.126854] 		ref#0: shared data backref parent 492981370880 count 1
> > > [26062.126856] 		ref#1: shared data backref parent 438017720320 count 1
> > > [26062.126858] 	item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50
> > > [26062.126859] 		extent refs 2 gen 2369347 flags 1
> > > [26062.126860] 		ref#0: shared data backref parent 492981370880 count 1
> > > [26062.126862] 		ref#1: shared data backref parent 438017720320 count 1
> > > [26062.126864] 	item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50
> > > [26062.126865] 		extent refs 2 gen 2369347 flags 1
> > > [26062.126866] 		ref#0: shared data backref parent 492981370880 count 1
> > > [26062.126868] 		ref#1: shared data backref parent 438017720320 count 1
> > > [26062.126870] 	item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50
> > > [26062.126871] 		extent refs 2 gen 2369347 flags 1
> > > [26062.126873] 		ref#0: shared data backref parent 492981370880 count 1
> > > [26062.126874] 		ref#1: shared data backref parent 438017720320 count 1
> > > [26062.126876] 	item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50
> > > [26062.126877] 		extent refs 2 gen 2369347 flags 1
> > > [26062.126878] 		ref#0: shared data backref parent 492981370880 count 1
> > > [26062.126880] 		ref#1: shared data backref parent 438017720320 count 1
> > > [26062.126882] 	item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50
> > > [26062.126883] 		extent refs 2 gen 2369347 flags 1
> > > [26062.126884] 		ref#0: shared data backref parent 492981370880 count 1
> > > [26062.126885] 		ref#1: shared data backref parent 438017720320 count 1
> > > [26062.126888] 	item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50
> > > [26062.126889] 		extent refs 2 gen 2369347 flags 1
> > > [26062.126890] 		ref#0: shared data backref parent 492981370880 count 1
> > > [26062.126891] 		ref#1: shared data backref parent 438017720320 count 1
> > > [26062.126893] 	item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79
> > > [26062.126894] 		extent refs 3 gen 2369347 flags 1
> > > [26062.126895] 		ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1
> > > [26062.126897] 		ref#1: shared data backref parent 492981370880 count 1
> > > [26062.126898] 		ref#2: shared data backref parent 438017720320 count 1
> > > [26062.126900] 	item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66
> > > [26062.126902] 		extent refs 2 gen 2369347 flags 1
> > > [26062.126903] 		ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1
> > > [26062.126904] 		ref#1: shared data backref parent 438017720320 count 1
> > > [26062.126906] 	item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37
> > > [26062.126907] 		extent refs 1 gen 2369352 flags 1
> > > [26062.126908] 		ref#0: shared data backref parent 510058790912 count 1
> > > [26062.126910] 	item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79
> > > [26062.126912] 		extent refs 6 gen 1800750 flags 1
> > > [26062.126913] 		ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2
> > > [26062.126915] 		ref#1: shared data backref parent 509782933504 count 2
> > > [26062.126917] 		ref#2: shared data backref parent 508543795200 count 2
> > > [26062.126919] 	item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66
> > > [26062.126920] 		extent refs 2 gen 2369075 flags 1
> > > [26062.126921] 		ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1
> > > [26062.126923] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.126925] 	item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79
> > > [26062.126926] 		extent refs 3 gen 1800733 flags 1
> > > [26062.126927] 		ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1
> > > [26062.126929] 		ref#1: shared data backref parent 509783719936 count 1
> > > [26062.126930] 		ref#2: shared data backref parent 509351231488 count 1
> > > [26062.126932] 	item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66
> > > [26062.126934] 		extent refs 2 gen 2420471 flags 1
> > > [26062.126934] 		ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1
> > > [26062.126936] 		ref#1: shared data backref parent 41020112896 count 1
> > > [26062.126938] 	item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66
> > > [26062.126939] 		extent refs 2 gen 2420471 flags 1
> > > [26062.126940] 		ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1
> > > [26062.126943] 		ref#1: shared data backref parent 41020112896 count 1
> > > [26062.126945] 	item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79
> > > [26062.126946] 		extent refs 3 gen 1800733 flags 1
> > > [26062.126947] 		ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1
> > > [26062.126949] 		ref#1: shared data backref parent 509783719936 count 1
> > > [26062.126950] 		ref#2: shared data backref parent 509351231488 count 1
> > > [26062.126953] 	item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66
> > > [26062.126954] 		extent refs 2 gen 2420502 flags 1
> > > [26062.126955] 		ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1
> > > [26062.126957] 		ref#1: shared data backref parent 492274728960 count 1
> > > [26062.126959] 	item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66
> > > [26062.126961] 		extent refs 2 gen 2420478 flags 1
> > > [26062.126962] 		ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1
> > > [26062.126964] 		ref#1: shared data backref parent 41020112896 count 1
> > > [26062.126966] 	item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37
> > > [26062.126967] 		extent refs 1 gen 1444854 flags 1
> > > [26062.126968] 		ref#0: shared data backref parent 214908928 count 1
> > > [26062.126970] 	item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53
> > > [26062.126971] 		extent refs 1 gen 2376479 flags 1
> > > [26062.126972] 		ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1
> > > [26062.126975] 	item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53
> > > [26062.126976] 		extent refs 1 gen 1444854 flags 1
> > > [26062.126977] 		ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1
> > > [26062.126979] 	item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37
> > > [26062.126981] 		extent refs 1 gen 1444854 flags 1
> > > [26062.126982] 		ref#0: shared data backref parent 93732864 count 1
> > > [26062.126984] 	item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37
> > > [26062.126985] 		extent refs 1 gen 1444854 flags 1
> > > [26062.126986] 		ref#0: shared data backref parent 93732864 count 1
> > > [26062.126988] 	item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37
> > > [26062.126990] 		extent refs 1 gen 1444854 flags 1
> > > [26062.126991] 		ref#0: shared data backref parent 93732864 count 1
> > > [26062.126993] 	item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37
> > > [26062.126994] 		extent refs 1 gen 1444854 flags 1
> > > [26062.126995] 		ref#0: shared data backref parent 214908928 count 1
> > > [26062.126997] 	item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92
> > > [26062.126998] 		extent refs 4 gen 2419085 flags 1
> > > [26062.126999] 		ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1
> > > [26062.127001] 		ref#1: shared data backref parent 494075035648 count 1
> > > [26062.127002] 		ref#2: shared data backref parent 470850240512 count 1
> > > [26062.127004] 		ref#3: shared data backref parent 437034745856 count 1
> > > [26062.127006] 	item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79
> > > [26062.127008] 		extent refs 3 gen 1800733 flags 1
> > > [26062.127009] 		ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1
> > > [26062.127011] 		ref#1: shared data backref parent 509783719936 count 1
> > > [26062.127013] 		ref#2: shared data backref parent 509351231488 count 1
> > > [26062.127015] 	item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66
> > > [26062.127016] 		extent refs 2 gen 2368859 flags 1
> > > [26062.127017] 		ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1
> > > [26062.127019] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.127021] 	item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53
> > > [26062.127023] 		extent refs 1 gen 2376479 flags 1
> > > [26062.127024] 		ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1
> > > [26062.127026] 	item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183
> > > [26062.127028] 		extent refs 11 gen 1800674 flags 1
> > > [26062.127029] 		ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1
> > > [26062.127031] 		ref#1: shared data backref parent 508153118720 count 1
> > > [26062.127032] 		ref#2: shared data backref parent 508031533056 count 1
> > > [26062.127034] 		ref#3: shared data backref parent 493564477440 count 1
> > > [26062.127036] 		ref#4: shared data backref parent 493540179968 count 1
> > > [26062.127038] 		ref#5: shared data backref parent 492675366912 count 1
> > > [26062.127039] 		ref#6: shared data backref parent 475132116992 count 1
> > > [26062.127041] 		ref#7: shared data backref parent 471032922112 count 1
> > > [26062.127042] 		ref#8: shared data backref parent 436112244736 count 1
> > > [26062.127044] 		ref#9: shared data backref parent 883998720 count 1
> > > [26062.127046] 		ref#10: shared data backref parent 87015424 count 1
> > > [26062.127048] 	item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37
> > > [26062.127050] 		extent refs 1 gen 2408131 flags 1
> > > [26062.127052] 		ref#0: shared data backref parent 509278633984 count 1
> > > [26062.127055] 	item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183
> > > [26062.127056] 		extent refs 11 gen 1800715 flags 1
> > > [26062.127058] 		ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1
> > > [26062.127059] 		ref#1: shared data backref parent 508152889344 count 1
> > > [26062.127061] 		ref#2: shared data backref parent 508031139840 count 1
> > > [26062.127062] 		ref#3: shared data backref parent 493563478016 count 1
> > > [26062.127065] 		ref#4: shared data backref parent 493539983360 count 1
> > > [26062.127066] 		ref#5: shared data backref parent 492674990080 count 1
> > > [26062.127068] 		ref#6: shared data backref parent 475132166144 count 1
> > > [26062.127070] 		ref#7: shared data backref parent 471032348672 count 1
> > > [26062.127071] 		ref#8: shared data backref parent 436111769600 count 1
> > > [26062.127073] 		ref#9: shared data backref parent 883359744 count 1
> > > [26062.127074] 		ref#10: shared data backref parent 85082112 count 1
> > > [26062.127077] 	item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92
> > > [26062.127078] 		extent refs 4 gen 2419085 flags 1
> > > [26062.127079] 		ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1
> > > [26062.127085] 		ref#1: shared data backref parent 494075035648 count 1
> > > [26062.127088] 		ref#2: shared data backref parent 470850240512 count 1
> > > [26062.127089] 		ref#3: shared data backref parent 437034745856 count 1
> > > [26062.127093] 	item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66
> > > [26062.127095] 		extent refs 2 gen 2421893 flags 1
> > > [26062.127096] 		ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1
> > > [26062.127099] 		ref#1: shared data backref parent 493703069696 count 1
> > > [26062.127102] 	item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53
> > > [26062.127104] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127105] 		ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1
> > > [26062.127109] 	item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66
> > > [26062.127111] 		extent refs 2 gen 2368700 flags 1
> > > [26062.127112] 		ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1
> > > [26062.127114] 		ref#1: shared data backref parent 494105804800 count 1
> > > [26062.127118] 	item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37
> > > [26062.127120] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127121] 		ref#0: shared data backref parent 93732864 count 1
> > > [26062.127125] 	item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37
> > > [26062.127127] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127128] 		ref#0: shared data backref parent 214433792 count 1
> > > [26062.127131] 	item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37
> > > [26062.127132] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127133] 		ref#0: shared data backref parent 214908928 count 1
> > > [26062.127136] 	item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79
> > > [26062.127138] 		extent refs 3 gen 1800750 flags 1
> > > [26062.127139] 		ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1
> > > [26062.127141] 		ref#1: shared data backref parent 509783556096 count 1
> > > [26062.127143] 		ref#2: shared data backref parent 508544974848 count 1
> > > [26062.127147] 	item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66
> > > [26062.127148] 		extent refs 2 gen 2368707 flags 1
> > > [26062.127149] 		ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1
> > > [26062.127151] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.127153] 	item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183
> > > [26062.127155] 		extent refs 11 gen 1800674 flags 1
> > > [26062.127156] 		ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1
> > > [26062.127157] 		ref#1: shared data backref parent 508152905728 count 1
> > > [26062.127159] 		ref#2: shared data backref parent 508031057920 count 1
> > > [26062.127160] 		ref#3: shared data backref parent 493563166720 count 1
> > > [26062.127161] 		ref#4: shared data backref parent 493539934208 count 1
> > > [26062.127163] 		ref#5: shared data backref parent 492674908160 count 1
> > > [26062.127164] 		ref#6: shared data backref parent 475884208128 count 1
> > > [26062.127165] 		ref#7: shared data backref parent 471032266752 count 1
> > > [26062.127167] 		ref#8: shared data backref parent 436111720448 count 1
> > > [26062.127168] 		ref#9: shared data backref parent 883179520 count 1
> > > [26062.127170] 		ref#10: shared data backref parent 83017728 count 1
> > > [26062.127172] 	item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37
> > > [26062.127173] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127174] 		ref#0: shared data backref parent 214908928 count 1
> > > [26062.127176] 	item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66
> > > [26062.127177] 		extent refs 2 gen 2381729 flags 1
> > > [26062.127178] 		ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1
> > > [26062.127180] 		ref#1: shared data backref parent 437808545792 count 1
> > > [26062.127182] 	item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66
> > > [26062.127183] 		extent refs 2 gen 2369098 flags 1
> > > [26062.127184] 		ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1
> > > [26062.127186] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.127188] 	item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66
> > > [26062.127189] 		extent refs 2 gen 2368700 flags 1
> > > [26062.127190] 		ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1
> > > [26062.127191] 		ref#1: shared data backref parent 494105804800 count 1
> > > [26062.127193] 	item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37
> > > [26062.127195] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127195] 		ref#0: shared data backref parent 102547456 count 1
> > > [26062.127197] 	item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79
> > > [26062.127198] 		extent refs 3 gen 1800750 flags 1
> > > [26062.127199] 		ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1
> > > [26062.127201] 		ref#1: shared data backref parent 509783408640 count 1
> > > [26062.127202] 		ref#2: shared data backref parent 508543451136 count 1
> > > [26062.127204] 	item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37
> > > [26062.127205] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127206] 		ref#0: shared data backref parent 93732864 count 1
> > > [26062.127208] 	item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37
> > > [26062.127209] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127210] 		ref#0: shared data backref parent 93732864 count 1
> > > [26062.127213] 	item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53
> > > [26062.127214] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127215] 		ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1
> > > [26062.127217] 	item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92
> > > [26062.127219] 		extent refs 4 gen 2419085 flags 1
> > > [26062.127220] 		ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1
> > > [26062.127222] 		ref#1: shared data backref parent 494075035648 count 1
> > > [26062.127223] 		ref#2: shared data backref parent 470850240512 count 1
> > > [26062.127225] 		ref#3: shared data backref parent 437034745856 count 1
> > > [26062.127226] 	item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79
> > > [26062.127228] 		extent refs 3 gen 1800750 flags 1
> > > [26062.127229] 		ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1
> > > [26062.127230] 		ref#1: shared data backref parent 509783408640 count 1
> > > [26062.127232] 		ref#2: shared data backref parent 508543451136 count 1
> > > [26062.127234] 	item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66
> > > [26062.127235] 		extent refs 2 gen 2368731 flags 1
> > > [26062.127236] 		ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1
> > > [26062.127237] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.127239] 	item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37
> > > [26062.127241] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127241] 		ref#0: shared data backref parent 93732864 count 1
> > > [26062.127243] 	item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37
> > > [26062.127244] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127245] 		ref#0: shared data backref parent 93732864 count 1
> > > [26062.127247] 	item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37
> > > [26062.127249] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127249] 		ref#0: shared data backref parent 93732864 count 1
> > > [26062.127252] 	item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37
> > > [26062.127253] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127254] 		ref#0: shared data backref parent 93732864 count 1
> > > [26062.127257] 	item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37
> > > [26062.127258] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127259] 		ref#0: shared data backref parent 93732864 count 1
> > > [26062.127262] 	item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53
> > > [26062.127263] 		extent refs 1 gen 2381741 flags 1
> > > [26062.127264] 		ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1
> > > [26062.127266] 	item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66
> > > [26062.127268] 		extent refs 2 gen 2368882 flags 1
> > > [26062.127269] 		ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1
> > > [26062.127270] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.127272] 	item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66
> > > [26062.127274] 		extent refs 2 gen 2368909 flags 1
> > > [26062.127275] 		ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1
> > > [26062.127277] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.127279] 	item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66
> > > [26062.127280] 		extent refs 2 gen 2369146 flags 1
> > > [26062.127281] 		ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1
> > > [26062.127283] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.127285] 	item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37
> > > [26062.127286] 		extent refs 1 gen 2369353 flags 1
> > > [26062.127287] 		ref#0: shared data backref parent 510167498752 count 1
> > > [26062.127289] 	item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79
> > > [26062.127294] 		extent refs 3 gen 1800733 flags 1
> > > [26062.127294] 		ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1
> > > [26062.127296] 		ref#1: shared data backref parent 509783752704 count 1
> > > [26062.127298] 		ref#2: shared data backref parent 509351313408 count 1
> > > [26062.127299] 	item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183
> > > [26062.127303] 		extent refs 22 gen 1800674 flags 1
> > > [26062.127304] 		ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2
> > > [26062.127306] 		ref#1: shared data backref parent 508152889344 count 2
> > > [26062.127307] 		ref#2: shared data backref parent 508031139840 count 2
> > > [26062.127308] 		ref#3: shared data backref parent 493563478016 count 2
> > > [26062.127310] 		ref#4: shared data backref parent 493539983360 count 2
> > > [26062.127311] 		ref#5: shared data backref parent 492674990080 count 2
> > > [26062.127312] 		ref#6: shared data backref parent 475132166144 count 2
> > > [26062.127314] 		ref#7: shared data backref parent 471032348672 count 2
> > > [26062.127315] 		ref#8: shared data backref parent 436111769600 count 2
> > > [26062.127317] 		ref#9: shared data backref parent 883359744 count 2
> > > [26062.127318] 		ref#10: shared data backref parent 85082112 count 2
> > > [26062.127320] 	item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183
> > > [26062.127321] 		extent refs 33 gen 1800674 flags 1
> > > [26062.127322] 		ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3
> > > [26062.127324] 		ref#1: shared data backref parent 508152889344 count 3
> > > [26062.127325] 		ref#2: shared data backref parent 508031139840 count 3
> > > [26062.127327] 		ref#3: shared data backref parent 493563478016 count 3
> > > [26062.127328] 		ref#4: shared data backref parent 493539983360 count 3
> > > [26062.127329] 		ref#5: shared data backref parent 492674990080 count 3
> > > [26062.127331] 		ref#6: shared data backref parent 475132166144 count 3
> > > [26062.127333] 		ref#7: shared data backref parent 471032348672 count 3
> > > [26062.127334] 		ref#8: shared data backref parent 436111769600 count 3
> > > [26062.127335] 		ref#9: shared data backref parent 883359744 count 3
> > > [26062.127337] 		ref#10: shared data backref parent 85082112 count 3
> > > [26062.127338] 	item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92
> > > [26062.127340] 		extent refs 4 gen 2419085 flags 1
> > > [26062.127341] 		ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1
> > > [26062.127342] 		ref#1: shared data backref parent 494075035648 count 1
> > > [26062.127344] 		ref#2: shared data backref parent 470850240512 count 1
> > > [26062.127345] 		ref#3: shared data backref parent 437034745856 count 1
> > > [26062.127347] 	item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66
> > > [26062.127348] 		extent refs 2 gen 2368932 flags 1
> > > [26062.127349] 		ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1
> > > [26062.127351] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.127353] 	item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37
> > > [26062.127354] 		extent refs 1 gen 2369353 flags 1
> > > [26062.127355] 		ref#0: shared data backref parent 510165155840 count 1
> > > [26062.127356] 	item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66
> > > [26062.127358] 		extent refs 2 gen 2369168 flags 1
> > > [26062.127359] 		ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1
> > > [26062.127360] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.127362] 	item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53
> > > [26062.127364] 		extent refs 1 gen 2369305 flags 1
> > > [26062.127365] 		ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1
> > > [26062.127367] 	item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92
> > > [26062.127369] 		extent refs 4 gen 2419085 flags 1
> > > [26062.127370] 		ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1
> > > [26062.127372] 		ref#1: shared data backref parent 494075035648 count 1
> > > [26062.127374] 		ref#2: shared data backref parent 470850240512 count 1
> > > [26062.127375] 		ref#3: shared data backref parent 437034745856 count 1
> > > [26062.127377] 	item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66
> > > [26062.127378] 		extent refs 2 gen 2368955 flags 1
> > > [26062.127379] 		ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1
> > > [26062.127382] 		ref#1: shared data backref parent 437615230976 count 1
> > > [26062.127384] 	item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37
> > > [26062.127385] 		extent refs 1 gen 2369352 flags 1
> > > [26062.127386] 		ref#0: shared data backref parent 510178295808 count 1
> > > [26062.127388] 	item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37
> > > [26062.127390] 		extent refs 1 gen 2369352 flags 1
> > > [26062.127391] 		ref#0: shared data backref parent 510058790912 count 1
> > > [26062.127393] 	item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37
> > > [26062.127394] 		extent refs 1 gen 2369352 flags 1
> > > [26062.127395] 		ref#0: shared data backref parent 510059184128 count 1
> > > [26062.127398] 	item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53
> > > [26062.127399] 		extent refs 1 gen 2378552 flags 1
> > > [26062.127400] 		ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1
> > > [26062.127403] 	item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53
> > > [26062.127405] 		extent refs 1 gen 2378552 flags 1
> > > [26062.127406] 		ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1
> > > [26062.127408] 	item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53
> > > [26062.127409] 		extent refs 1 gen 2381741 flags 1
> > > [26062.127410] 		ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1
> > > [26062.127413] 	item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66
> > > [26062.127414] 		extent refs 2 gen 2368700 flags 1
> > > [26062.127415] 		ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1
> > > [26062.127416] 		ref#1: shared data backref parent 494105804800 count 1
> > > [26062.127419] 	item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66
> > > [26062.127420] 		extent refs 2 gen 2368700 flags 1
> > > [26062.127421] 		ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1
> > > [26062.127423] 		ref#1: shared data backref parent 494105804800 count 1
> > > [26062.127425] 	item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66
> > > [26062.127426] 		extent refs 2 gen 2368700 flags 1
> > > [26062.127428] 		ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1
> > > [26062.127430] 		ref#1: shared data backref parent 494105804800 count 1
> > > [26062.127432] 	item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183
> > > [26062.127434] 		extent refs 11 gen 1800699 flags 1
> > > [26062.127435] 		ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1
> > > [26062.127437] 		ref#1: shared data backref parent 508152987648 count 1
> > > [26062.127439] 		ref#2: shared data backref parent 508031172608 count 1
> > > [26062.127440] 		ref#3: shared data backref parent 493563822080 count 1
> > > [26062.127441] 		ref#4: shared data backref parent 493540130816 count 1
> > > [26062.127443] 		ref#5: shared data backref parent 492675170304 count 1
> > > [26062.127444] 		ref#6: shared data backref parent 475884584960 count 1
> > > [26062.127446] 		ref#7: shared data backref parent 471032496128 count 1
> > > [26062.127447] 		ref#8: shared data backref parent 436111884288 count 1
> > > [26062.127448] 		ref#9: shared data backref parent 883703808 count 1
> > > [26062.127450] 		ref#10: shared data backref parent 85671936 count 1
> > > [26062.127452] 	item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37
> > > [26062.127453] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127454] 		ref#0: shared data backref parent 214908928 count 1
> > > [26062.127457] 	item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37
> > > [26062.127458] 		extent refs 1 gen 2426255 flags 1
> > > [26062.127459] 		ref#0: shared data backref parent 436271005696 count 1
> > > [26062.127461] 	item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183
> > > [26062.127463] 		extent refs 22 gen 1800674 flags 1
> > > [26062.127464] 		ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2
> > > [26062.127465] 		ref#1: shared data backref parent 508152987648 count 2
> > > [26062.127467] 		ref#2: shared data backref parent 508031172608 count 2
> > > [26062.127468] 		ref#3: shared data backref parent 493563822080 count 2
> > > [26062.127470] 		ref#4: shared data backref parent 493540130816 count 2
> > > [26062.127472] 		ref#5: shared data backref parent 492675170304 count 2
> > > [26062.127473] 		ref#6: shared data backref parent 475884584960 count 2
> > > [26062.127475] 		ref#7: shared data backref parent 471032496128 count 2
> > > [26062.127476] 		ref#8: shared data backref parent 436111884288 count 2
> > > [26062.127478] 		ref#9: shared data backref parent 883703808 count 2
> > > [26062.127480] 		ref#10: shared data backref parent 85671936 count 2
> > > [26062.127483] 	item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37
> > > [26062.127486] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127488] 		ref#0: shared data backref parent 167198720 count 1
> > > [26062.127491] 	item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66
> > > [26062.127494] 		extent refs 2 gen 1400856 flags 1
> > > [26062.127496] 		ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1
> > > [26062.127498] 		ref#1: shared data backref parent 470954196992 count 1
> > > [26062.127501] 	item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37
> > > [26062.127503] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127504] 		ref#0: shared data backref parent 93732864 count 1
> > > [26062.127507] 	item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53
> > > [26062.127510] 		extent refs 1 gen 2381741 flags 1
> > > [26062.127512] 		ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1
> > > [26062.127518] 	item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66
> > > [26062.127520] 		extent refs 2 gen 2368701 flags 1
> > > [26062.127521] 		ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1
> > > [26062.127523] 		ref#1: shared data backref parent 494107131904 count 1
> > > [26062.127526] 	item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183
> > > [26062.127527] 		extent refs 44 gen 1800674 flags 1
> > > [26062.127528] 		ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4
> > > [26062.127530] 		ref#1: shared data backref parent 508152987648 count 4
> > > [26062.127532] 		ref#2: shared data backref parent 508031172608 count 4
> > > [26062.127533] 		ref#3: shared data backref parent 493563822080 count 4
> > > [26062.127535] 		ref#4: shared data backref parent 493540130816 count 4
> > > [26062.127537] 		ref#5: shared data backref parent 492675170304 count 4
> > > [26062.127539] 		ref#6: shared data backref parent 475884584960 count 4
> > > [26062.127540] 		ref#7: shared data backref parent 471032496128 count 4
> > > [26062.127542] 		ref#8: shared data backref parent 436111884288 count 4
> > > [26062.127543] 		ref#9: shared data backref parent 883703808 count 4
> > > [26062.127545] 		ref#10: shared data backref parent 85671936 count 4
> > > [26062.127547] 	item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79
> > > [26062.127549] 		extent refs 3 gen 1800750 flags 1
> > > [26062.127552] 		ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1
> > > [26062.127554] 		ref#1: shared data backref parent 509783130112 count 1
> > > [26062.127557] 		ref#2: shared data backref parent 508544876544 count 1
> > > [26062.127560] 	item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37
> > > [26062.127561] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127562] 		ref#0: shared data backref parent 93732864 count 1
> > > [26062.127565] 	item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37
> > > [26062.127566] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127568] 		ref#0: shared data backref parent 214908928 count 1
> > > [26062.127570] 	item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37
> > > [26062.127572] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127573] 		ref#0: shared data backref parent 93732864 count 1
> > > [26062.127577] 	item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24
> > > [26062.127580] 		block group used 834060288 chunk_objectid 256 flags 1
> > > [26062.127581] 	item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37
> > > [26062.127583] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127584] 		ref#0: shared data backref parent 167198720 count 1
> > > [26062.127586] 	item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37
> > > [26062.127587] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127588] 		ref#0: shared data backref parent 167198720 count 1
> > > [26062.127590] 	item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37
> > > [26062.127591] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127592] 		ref#0: shared data backref parent 167198720 count 1
> > > [26062.127594] 	item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37
> > > [26062.127595] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127596] 		ref#0: shared data backref parent 167198720 count 1
> > > [26062.127598] 	item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37
> > > [26062.127599] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127600] 		ref#0: shared data backref parent 167198720 count 1
> > > [26062.127602] 	item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37
> > > [26062.127603] 		extent refs 1 gen 1444854 flags 1
> > > [26062.127604] 		ref#0: shared data backref parent 167198720 count 1
> > > [26062.127606] 	item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183
> > > [26062.127607] 		extent refs 11 gen 2409230 flags 1
> > > [26062.127608] 		ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1
> > > [26062.127610] 		ref#1: shared data backref parent 508153675776 count 1
> > > [26062.127611] 		ref#2: shared data backref parent 493566640128 count 1
> > > [26062.127613] 		ref#3: shared data backref parent 493540999168 count 1
> > > [26062.127614] 		ref#4: shared data backref parent 492676497408 count 1
> > > [26062.127616] 		ref#5: shared data backref parent 475884634112 count 1
> > > [26062.127617] 		ref#6: shared data backref parent 475691106304 count 1
> > > [26062.127619] 		ref#7: shared data backref parent 471034494976 count 1
> > > [26062.127620] 		ref#8: shared data backref parent 436113883136 count 1
> > > [26062.127622] 		ref#9: shared data backref parent 886407168 count 1
> > > [26062.127623] 		ref#10: shared data backref parent 89440256 count 1
> > > [26062.127625] 	item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170
> > > [26062.127626] 		extent refs 10 gen 2413323 flags 1
> > > [26062.127629] 		ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1
> > > [26062.127631] 		ref#1: shared data backref parent 508153675776 count 1
> > > [26062.127633] 		ref#2: shared data backref parent 493566640128 count 1
> > > [26062.127635] 		ref#3: shared data backref parent 493540999168 count 1
> > > [26062.127636] 		ref#4: shared data backref parent 492676497408 count 1
> > > [26062.127638] 		ref#5: shared data backref parent 475691106304 count 1
> > > [26062.127640] 		ref#6: shared data backref parent 471034494976 count 1
> > > [26062.127642] 		ref#7: shared data backref parent 436113883136 count 1
> > > [26062.127643] 		ref#8: shared data backref parent 886407168 count 1
> > > [26062.127645] 		ref#9: shared data backref parent 89440256 count 1
> > > [26062.127647] 	item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37
> > > [26062.127648] 		extent refs 1 gen 2407793 flags 1
> > > [26062.127649] 		ref#0: shared data backref parent 475884634112 count 1
> > > [26062.127653] 	item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183
> > > [26062.127655] 		extent refs 11 gen 1800729 flags 1
> > > [26062.127656] 		ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1
> > > [26062.127658] 		ref#1: shared data backref parent 508153069568 count 1
> > > [26062.127659] 		ref#2: shared data backref parent 508031270912 count 1
> > > [26062.127661] 		ref#3: shared data backref parent 493564215296 count 1
> > > [26062.127663] 		ref#4: shared data backref parent 493540098048 count 1
> > > [26062.127664] 		ref#5: shared data backref parent 492676038656 count 1
> > > [26062.127666] 		ref#6: shared data backref parent 475884142592 count 1
> > > [26062.127668] 		ref#7: shared data backref parent 471032840192 count 1
> > > [26062.127669] 		ref#8: shared data backref parent 436112146432 count 1
> > > [26062.127671] 		ref#9: shared data backref parent 883752960 count 1
> > > [26062.127674] 		ref#10: shared data backref parent 86654976 count 1
> > > [26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456  owner 73020573 offset 17039360
> > > [26062.127680] ------------[ cut here ]------------
> > > [26062.127682] BTRFS: Transaction aborted (error -2)
> > > [26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921
> > > [26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc s
 nd_hda_intel
> > > [26062.127733]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
> > > [26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
> > > [26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
> > > [26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921
> > > [26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df
> > > [26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282
> > > [26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007
> > > [26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550
> > > [26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
> > > [26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169
> > > [26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
> > > [26062.127815] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
> > > [26062.127817] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
> > > [26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > [26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > > [26062.127823] Call Trace:
> > > [26062.127832]  __btrfs_run_delayed_refs+0x750/0xc36
> > > [26062.127837]  ? __switch_to_asm+0x41/0x70
> > > [26062.127839]  ? __switch_to_asm+0x35/0x70
> > > [26062.127841]  ? __switch_to_asm+0x41/0x70
> > > [26062.127848]  ? __switch_to+0x13d/0x3d5
> > > [26062.127852]  btrfs_run_delayed_refs+0x5d/0x132
> > > [26062.127855]  btrfs_commit_transaction+0x55/0x7c8
> > > [26062.127858]  ? start_transaction+0x347/0x3cb
> > > [26062.127862]  transaction_kthread+0xc9/0x135
> > > [26062.127865]  ? btrfs_cleanup_transaction+0x403/0x403
> > > [26062.127869]  kthread+0xeb/0xf0
> > > [26062.127872]  ? kthread_create_worker_on_cpu+0x65/0x65
> > > [26062.127875]  ret_from_fork+0x35/0x40
> > > [26062.127879] ---[ end trace 4c1a6b3749a2f651 ]---
> > > [26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry
> > > [26062.127917] BTRFS info (device dm-2): forced readonly
> > > [26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entry

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing'
@ 2020-05-24 21:30       ` Marc MERLIN
  2020-05-25  3:01         ` Marc MERLIN
  2020-05-25 16:37         ` Chris Murphy
  0 siblings, 2 replies; 479+ messages in thread
From: Marc MERLIN @ 2020-05-24 21:30 UTC (permalink / raw)
  To: Su Yue, linux-btrfs, Qu Wenruo, Josef Bacik; +Cc: Su Yue

My data is fine, it's double backed up and the filesystem is still mountable without issues.
But I had an error that broke btrfs send, and after fixing it with repair, I'm stuck with thses 'csum missing'

Any idea if I can fix them without deleting the filesystem?

Current state:
saruman:~# btrfs check --repair /dev/mapper/cr
enabling repair mode
Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/mapper/cr
UUID: 4cb82363-e833-444e-b23e-1597a14a8aab
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
No device size related problem found
[3/7] checking free space cache
cache and super generation don't match, space cache will be invalidated
[4/7] checking fs roots
root 356 inode 14058737 errors 1000, some csum missing
root 197362 inode 14058737 errors 1000, some csum missing
root 199551 inode 14058737 errors 1000, some csum missing
(snip)
root 204167 inode 14058737 errors 1000, some csum missing
root 204168 inode 14058737 errors 1000, some csum missing
ERROR: errors found in fs roots
found 112392159232 bytes used, error(s) found
total csum bytes: 105763716
total tree bytes: 4036018176
total fs tree bytes: 3666313216
total extent tree bytes: 214728704
btree space waste bytes: 875560453
file data blocks allocated: 207903920128
 referenced 206918037504

I can run multiple times, 'some csum missing' does not get fixed.

Lowmem does not fix it either:
saruman:~# btrfs check --mode=lowmem --repair /dev/mapper/cr
Starting repair.
WARNING: low-memory mode repair support is only partial
Opening filesystem to check...
Checking filesystem on /dev/mapper/cr
UUID: 4cb82363-e833-444e-b23e-1597a14a8aab
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
No device size related problem found
[3/7] checking free space cache
cache and super generation don't match, space cache will be invalidated
[4/7] checking fs roots
ERROR: root 356 EXTENT_DATA[14058737 0] csum missing, have: 0, expected: 8192
ERROR: root 356 EXTENT_DATA[14058737 0] compressed extent must have csum, but only 0 bytes have, expect 8192
Add a hole [0, 4096] in inode [33037178]
Add a hole [0, 4096] in inode [33037180]
Add a hole [0, 4096] in inode [33037181]
Add a hole [0, 4096] in inode [33037188]
Add a hole [0, 4096] in inode [33037189]
Add a hole [0, 4096] in inode [33037191]
ERROR: root 197362 EXTENT_DATA[14058737 0] csum missing, have: 0, expected: 8192
ERROR: root 197362 EXTENT_DATA[14058737 0] compressed extent must have csum, but only 0 bytes have, expect 8192
Add a hole [0, 4096] in inode [33037178]
Add a hole [0, 4096] in inode [33037180]
(...)
Add a hole [0, 4096] in inode [33037189]
Add a hole [0, 4096] in inode [33037191]
ERROR: errors found in fs roots
found 112392294400 bytes used, error(s) found
total csum bytes: 105763716
total tree bytes: 4035985408
total fs tree bytes: 3666313216
total extent tree bytes: 214728704
btree space waste bytes: 875550920
file data blocks allocated: 207903920128
 referenced 206918037504


This originally started with:
BTRFS error (device dm-0): did not find backref in send_root. inode=14058737, offset=0, disk_byte=2694234112 found extent=2694234112

which I was able to fix with the repair below, but now I'm stuck with the 'some csum missing'.
Checking filesystem on /dev/mapper/cr
UUID: 4cb82363-e833-444e-b23e-1597a14a8aab
[1/7] checking root items
[2/7] checking extents
data backref 2694234112 root 356 owner 14058737 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 2694234112 root 356 owner 14058737 offset 0 found 1 wanted 0 back 0x55e7383a3a00
incorrect local backref count on 2694234112 root 2147484004 owner 14058737 offset 0 found 0 wanted 1 back 0x55e733d1c2b0
backref disk bytenr does not match extent record, bytenr=2694234112, ref bytenr=0
backpointer mismatch on [2694234112 8192]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
root 356 inode 33037179 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 8192
root 203332 inode 33037179 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 8192
root 203669 inode 33037179 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 8192
root 204006 inode 33037179 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 8192
root 204010 inode 33037179 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 8192
root 204024 inode 33037179 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 8192
root 204037 inode 33037179 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 8192
root 204049 inode 33037179 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 8192
root 204167 inode 33037179 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 8192
root 204168 inode 33037179 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 8192
ERROR: errors found in fs roots
found 111202623488 bytes used, error(s) found
total csum bytes: 105763728
total tree bytes: 2846048256
total fs tree bytes: 2485485568
total extent tree bytes: 205619200
btree space waste bytes: 668593131
file data blocks allocated: 149272375296
 referenced 140124020736
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing'
  2020-05-24 21:30       ` 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' Marc MERLIN
@ 2020-05-25  3:01         ` Marc MERLIN
  2020-05-25 16:37         ` Chris Murphy
  1 sibling, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2020-05-25  3:01 UTC (permalink / raw)
  To: Su Yue, linux-btrfs, Qu Wenruo, Josef Bacik; +Cc: Su Yue

I forgot to add:

I'll keep my system in the broken state for maybe a day or two at most,
but given that this is my root filesystem and that I'm booted off a
backup media, I can't stay in that state too long.

This may not be nothing new and if it's already known that the way out
is to wipe and restore backups, let me know.

Thnaks,
Marc

On Sun, May 24, 2020 at 02:30:59PM -0700, Marc MERLIN wrote:
> My data is fine, it's double backed up and the filesystem is still mountable without issues.
> But I had an error that broke btrfs send, and after fixing it with repair, I'm stuck with thses 'csum missing'
> 
> Any idea if I can fix them without deleting the filesystem?
> 
> Current state:
> saruman:~# btrfs check --repair /dev/mapper/cr
> enabling repair mode
> Starting repair.
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/cr
> UUID: 4cb82363-e833-444e-b23e-1597a14a8aab
> [1/7] checking root items
> Fixed 0 roots.
> [2/7] checking extents
> No device size related problem found
> [3/7] checking free space cache
> cache and super generation don't match, space cache will be invalidated
> [4/7] checking fs roots
> root 356 inode 14058737 errors 1000, some csum missing
> root 197362 inode 14058737 errors 1000, some csum missing
> root 199551 inode 14058737 errors 1000, some csum missing
> (snip)
> root 204167 inode 14058737 errors 1000, some csum missing
> root 204168 inode 14058737 errors 1000, some csum missing
> ERROR: errors found in fs roots
> found 112392159232 bytes used, error(s) found
> total csum bytes: 105763716
> total tree bytes: 4036018176
> total fs tree bytes: 3666313216
> total extent tree bytes: 214728704
> btree space waste bytes: 875560453
> file data blocks allocated: 207903920128
>  referenced 206918037504
> 
> I can run multiple times, 'some csum missing' does not get fixed.
> 
> Lowmem does not fix it either:
> saruman:~# btrfs check --mode=lowmem --repair /dev/mapper/cr
> Starting repair.
> WARNING: low-memory mode repair support is only partial
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/cr
> UUID: 4cb82363-e833-444e-b23e-1597a14a8aab
> [1/7] checking root items
> Fixed 0 roots.
> [2/7] checking extents
> No device size related problem found
> [3/7] checking free space cache
> cache and super generation don't match, space cache will be invalidated
> [4/7] checking fs roots
> ERROR: root 356 EXTENT_DATA[14058737 0] csum missing, have: 0, expected: 8192
> ERROR: root 356 EXTENT_DATA[14058737 0] compressed extent must have csum, but only 0 bytes have, expect 8192
> Add a hole [0, 4096] in inode [33037178]
> Add a hole [0, 4096] in inode [33037180]
> Add a hole [0, 4096] in inode [33037181]
> Add a hole [0, 4096] in inode [33037188]
> Add a hole [0, 4096] in inode [33037189]
> Add a hole [0, 4096] in inode [33037191]
> ERROR: root 197362 EXTENT_DATA[14058737 0] csum missing, have: 0, expected: 8192
> ERROR: root 197362 EXTENT_DATA[14058737 0] compressed extent must have csum, but only 0 bytes have, expect 8192
> Add a hole [0, 4096] in inode [33037178]
> Add a hole [0, 4096] in inode [33037180]
> (...)
> Add a hole [0, 4096] in inode [33037189]
> Add a hole [0, 4096] in inode [33037191]
> ERROR: errors found in fs roots
> found 112392294400 bytes used, error(s) found
> total csum bytes: 105763716
> total tree bytes: 4035985408
> total fs tree bytes: 3666313216
> total extent tree bytes: 214728704
> btree space waste bytes: 875550920
> file data blocks allocated: 207903920128
>  referenced 206918037504
> 
> 
> This originally started with:
> BTRFS error (device dm-0): did not find backref in send_root. inode=14058737, offset=0, disk_byte=2694234112 found extent=2694234112
> 
> which I was able to fix with the repair below, but now I'm stuck with the 'some csum missing'.
> Checking filesystem on /dev/mapper/cr
> UUID: 4cb82363-e833-444e-b23e-1597a14a8aab
> [1/7] checking root items
> [2/7] checking extents
> data backref 2694234112 root 356 owner 14058737 offset 0 num_refs 0 not found in extent tree
> incorrect local backref count on 2694234112 root 356 owner 14058737 offset 0 found 1 wanted 0 back 0x55e7383a3a00
> incorrect local backref count on 2694234112 root 2147484004 owner 14058737 offset 0 found 0 wanted 1 back 0x55e733d1c2b0
> backref disk bytenr does not match extent record, bytenr=2694234112, ref bytenr=0
> backpointer mismatch on [2694234112 8192]
> ERROR: errors found in extent allocation tree or chunk allocation
> [3/7] checking free space cache
> [4/7] checking fs roots
> root 356 inode 33037179 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 8192
> root 203332 inode 33037179 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 8192
> root 203669 inode 33037179 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 8192
> root 204006 inode 33037179 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 8192
> root 204010 inode 33037179 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 8192
> root 204024 inode 33037179 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 8192
> root 204037 inode 33037179 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 8192
> root 204049 inode 33037179 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 8192
> root 204167 inode 33037179 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 8192
> root 204168 inode 33037179 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 8192
> ERROR: errors found in fs roots
> found 111202623488 bytes used, error(s) found
> total csum bytes: 105763728
> total tree bytes: 2846048256
> total fs tree bytes: 2485485568
> total extent tree bytes: 205619200
> btree space waste bytes: 668593131
> file data blocks allocated: 149272375296
>  referenced 140124020736
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>  
> Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
> 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing'
  2020-05-24 21:30       ` 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' Marc MERLIN
  2020-05-25  3:01         ` Marc MERLIN
@ 2020-05-25 16:37         ` Chris Murphy
  2020-05-25 20:16           ` Marc MERLIN
  1 sibling, 1 reply; 479+ messages in thread
From: Chris Murphy @ 2020-05-25 16:37 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Su Yue, Btrfs BTRFS, Qu Wenruo, Josef Bacik, Su Yue

On Sun, May 24, 2020 at 3:31 PM Marc MERLIN <marc@merlins.org> wrote:
>
> My data is fine, it's double backed up and the filesystem is still mountable without issues.
> But I had an error that broke btrfs send, and after fixing it with repair, I'm stuck with thses 'csum missing'

I'm not following the sequence of events. The send|receive failed? Did
you try deleting the failed received snapshot?


> which I was able to fix with the repair below, but now I'm stuck with the 'some csum missing'.
> Checking filesystem on /dev/mapper/cr
> UUID: 4cb82363-e833-444e-b23e-1597a14a8aab
> [1/7] checking root items
> [2/7] checking extents
> data backref 2694234112 root 356 owner 14058737 offset 0 num_refs 0 not found in extent tree
> incorrect local backref count on 2694234112 root 356 owner 14058737 offset 0 found 1 wanted 0 back 0x55e7383a3a00
> incorrect local backref count on 2694234112 root 2147484004 owner 14058737 offset 0 found 0 wanted 1 back 0x55e733d1c2b0
> backref disk bytenr does not match extent record, bytenr=2694234112, ref bytenr=0
> backpointer mismatch on [2694234112 8192]
> ERROR: errors found in extent allocation tree or chunk allocation
> [3/7] checking free space cache
> [4/7] checking fs roots
> root 356 inode 33037179 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 8192
> root 203332 inode 33037179 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 8192


Is no-holes enabled on either file system?


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing'
  2020-05-25 16:37         ` Chris Murphy
@ 2020-05-25 20:16           ` Marc MERLIN
  2020-05-25 20:24             ` Chris Murphy
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2020-05-25 20:16 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Su Yue, Btrfs BTRFS, Qu Wenruo, Josef Bacik, Su Yue

On Mon, May 25, 2020 at 10:37:58AM -0600, Chris Murphy wrote:
> On Sun, May 24, 2020 at 3:31 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > My data is fine, it's double backed up and the filesystem is still mountable without issues.
> > But I had an error that broke btrfs send, and after fixing it with repair, I'm stuck with thses 'csum missing'
> 
> I'm not following the sequence of events. The send|receive failed? Did
> you try deleting the failed received snapshot?
 
btrfs send failed because there was an error on the source filesystem.
Then I found this in the logs
BTRFS error (device dm-0): did not find backref in send_root.  inode=14058737, offset=0, disk_byte=2694234112 found extent=2694234112

then I ran the btrfs check repair, with and without lowmem.
They fixed some things, but leave me 

> Is no-holes enabled on either file system?

Not intentionally. How do I check?

Either way, things happened, not exactly sure what, but I'm just trying
to find out if check --repair can/should do a better job, and if I can
give anything useful before wiping the filesystem and starting over.

As of right now, I have:
aruman:~# btrfs check /dev/mapper/cr
Opening filesystem to check...
Checking filesystem on /dev/mapper/cr
UUID: 4cb82363-e833-444e-b23e-1597a14a8aab
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
there is no free space entry for 24565465088-24565481472
there is no free space entry for 24565465088-24717033472
cache appears valid but isn't 23643291648
there is no free space entry for 45118177280-45118193664
there is no free space entry for 45118177280-46191869952
cache appears valid but isn't 45118128128
wanted offset 1659663106048, found 1659663089664
wanted offset 1659663106048, found 1659663089664
cache appears valid but isn't 1659663024128
[4/7] checking fs roots
root 356 inode 14058737 errors 1000, some csum missing
root 197362 inode 14058737 errors 1000, some csum missing
root 199551 inode 14058737 errors 1000, some csum missing
root 201742 inode 14058737 errors 1000, some csum missing
root 202370 inode 14058737 errors 1000, some csum missing
root 202683 inode 14058737 errors 1000, some csum missing
root 202999 inode 14058737 errors 1000, some csum missing
root 203332 inode 14058737 errors 1000, some csum missing
root 203669 inode 14058737 errors 1000, some csum missing
root 204006 inode 14058737 errors 1000, some csum missing
root 204010 inode 14058737 errors 1000, some csum missing
root 204024 inode 14058737 errors 1000, some csum missing
root 204037 inode 14058737 errors 1000, some csum missing
root 204049 inode 14058737 errors 1000, some csum missing
root 204063 inode 14058737 errors 1000, some csum missing
root 204077 inode 14058737 errors 1000, some csum missing
root 204091 inode 14058737 errors 1000, some csum missing
root 204105 inode 14058737 errors 1000, some csum missing
root 204119 inode 14058737 errors 1000, some csum missing
root 204133 inode 14058737 errors 1000, some csum missing
root 204146 inode 14058737 errors 1000, some csum missing
root 204147 inode 14058737 errors 1000, some csum missing
root 204159 inode 14058737 errors 1000, some csum missing
root 204160 inode 14058737 errors 1000, some csum missing
root 204161 inode 14058737 errors 1000, some csum missing
root 204162 inode 14058737 errors 1000, some csum missing
root 204163 inode 14058737 errors 1000, some csum missing
root 204164 inode 14058737 errors 1000, some csum missing
root 204165 inode 14058737 errors 1000, some csum missing
root 204166 inode 14058737 errors 1000, some csum missing
root 204167 inode 14058737 errors 1000, some csum missing
root 204168 inode 14058737 errors 1000, some csum missing
ERROR: errors found in fs roots
found 112391372800 bytes used, error(s) found
total csum bytes: 105763716
total tree bytes: 4036018176
total fs tree bytes: 3666313216
total extent tree bytes: 214728704
btree space waste bytes: 875571363
file data blocks allocated: 207903133696
 referenced 20

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing'
  2020-05-25 20:16           ` Marc MERLIN
@ 2020-05-25 20:24             ` Chris Murphy
  2020-05-25 20:39               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Chris Murphy @ 2020-05-25 20:24 UTC (permalink / raw)
  To: Marc MERLIN
  Cc: Chris Murphy, Su Yue, Btrfs BTRFS, Qu Wenruo, Josef Bacik, Su Yue

On Mon, May 25, 2020 at 2:16 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 25, 2020 at 10:37:58AM -0600, Chris Murphy wrote:
> > On Sun, May 24, 2020 at 3:31 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > My data is fine, it's double backed up and the filesystem is still mountable without issues.
> > > But I had an error that broke btrfs send, and after fixing it with repair, I'm stuck with thses 'csum missing'
> >
> > I'm not following the sequence of events. The send|receive failed? Did
> > you try deleting the failed received snapshot?
>
> btrfs send failed because there was an error on the source filesystem.
> Then I found this in the logs
> BTRFS error (device dm-0): did not find backref in send_root.  inode=14058737, offset=0, disk_byte=2694234112 found extent=2694234112
>
> then I ran the btrfs check repair, with and without lowmem.
> They fixed some things, but leave me


OK I didn't understand that the problem is with only the sending file
system, not the receive file system. And also it sounds like the send
did not cause the problem, but it's somehow a pre-existing problem
that --repair isn't completely fixing up, or maybe is making different
(or worse).

So I guess the real question is what happened to this file system
before the send, that got it into this weird state.


>
> > Is no-holes enabled on either file system?
>
> Not intentionally. How do I check?

btrfs insp dump-s

It's not yet the default and can't be inadvertently enabled so chances
are it's the original holes implementation.

-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing'
  2020-05-25 20:24             ` Chris Murphy
@ 2020-05-25 20:39               ` Marc MERLIN
  2020-05-25 22:47                 ` Chris Murphy
  2020-07-07  3:55                 ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN
  0 siblings, 2 replies; 479+ messages in thread
From: Marc MERLIN @ 2020-05-25 20:39 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Su Yue, Btrfs BTRFS, Qu Wenruo, Josef Bacik, Su Yue

On Mon, May 25, 2020 at 02:24:03PM -0600, Chris Murphy wrote:
> OK I didn't understand that the problem is with only the sending file
> system, not the receive file system. And also it sounds like the send
> did not cause the problem, but it's somehow a pre-existing problem
> that --repair isn't completely fixing up, or maybe is making different
> (or worse).
 
Correct on all points.

> So I guess the real question is what happened to this file system
> before the send, that got it into this weird state.

That too, but honestly there are a lot of variables, and it feels like a
bit of wild goose chase. 

Basically it looked like the issues with the FS are pretty minor (I was
able to cp -av the entire data without any file error), but that btrfs
check --repair is unable to make it right, which will likely force me to
wipe and restore.
I know chedk is WIP, and that's why I'm providing feedback :)

> > > Is no-holes enabled on either file system?
> >
> > Not intentionally. How do I check?
> 
> btrfs insp dump-s
> 
> It's not yet the default and can't be inadvertently enabled so chances
> are it's the original holes implementation.

The filesystem was definitely created a while ago (2-3 years?)

I have been recently been playing with bees, but I'm reasonably sure I didn't run in on that filesystem
it lists support for "HOLE extents and btrfs no-holes feature"


saruman:~# btrfs insp dump-s /dev/mapper/cr
superblock: bytenr=65536, device=/dev/mapper/cr
---------------------------------------------------------
csum_type               0 (crc32c)
csum_size               4
csum                    0x34a9cfed [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    4cb82363-e833-444e-b23e-1597a14a8aab
metadata_uuid           4cb82363-e833-444e-b23e-1597a14a8aab
label                   btrfs_boot
generation              3801933
root                    1686707896320
sys_array_size          97
chunk_root_generation   3801931
root_level              1
chunk_root              1679158165504
chunk_root_level        1
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             161057079296
bytes_used              112391540736
sectorsize              4096
nodesize                16384
leafsize (deprecated)   16384
stripesize              4096
root_dir                6
num_devices             1
compat_flags            0x0
compat_ro_flags         0x0
incompat_flags          0x169
                        ( MIXED_BACKREF |
                          COMPRESS_LZO |
                          BIG_METADATA |
                          EXTENDED_IREF |
                          SKINNY_METADATA )
cache_generation        3801932
uuid_tree_generation    3801932
dev_item.uuid           924bcb26-9c4c-4106-b3ce-1bb66e66de1f
dev_item.fsid           4cb82363-e833-444e-b23e-1597a14a8aab [match]
dev_item.type           0
dev_item.total_bytes    161057079296
dev_item.bytes_used     136423931904
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing'
  2020-05-25 20:39               ` Marc MERLIN
@ 2020-05-25 22:47                 ` Chris Murphy
  2020-05-25 22:51                   ` Chris Murphy
  2020-05-26  0:13                   ` Marc MERLIN
  2020-07-07  3:55                 ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN
  1 sibling, 2 replies; 479+ messages in thread
From: Chris Murphy @ 2020-05-25 22:47 UTC (permalink / raw)
  To: Marc MERLIN
  Cc: Chris Murphy, Su Yue, Btrfs BTRFS, Qu Wenruo, Josef Bacik, Su Yue

On Mon, May 25, 2020 at 2:39 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 25, 2020 at 02:24:03PM -0600, Chris Murphy wrote:
> > OK I didn't understand that the problem is with only the sending file
> > system, not the receive file system. And also it sounds like the send
> > did not cause the problem, but it's somehow a pre-existing problem
> > that --repair isn't completely fixing up, or maybe is making different
> > (or worse).
>
> Correct on all points.
>
> > So I guess the real question is what happened to this file system
> > before the send, that got it into this weird state.
>
> That too, but honestly there are a lot of variables, and it feels like a
> bit of wild goose chase.

Maybe. The story arc on Btrfs is that check --repair only fixes the
things it knows how to fix. It's gotten better but still has the scary
warning, and lately has a 10 second delay to really make sure the user
meant to use it. And regardless of mode, it's slow and just can't
scale. Neither does "wipe and restore from backup". So the problems of
inconsistency need to be understood to avoid the problem in the first
place.


>
> Basically it looked like the issues with the FS are pretty minor (I was
> able to cp -av the entire data without any file error), but that btrfs
> check --repair is unable to make it right, which will likely force me to
> wipe and restore.
> I know chedk is WIP, and that's why I'm providing feedback :)

What about finding inode 14058737 and deleting it? In all of the
listed subvolumes? And then unmount and check again?


> > > > Is no-holes enabled on either file system?
> > >
> > > Not intentionally. How do I check?
> >
> > btrfs insp dump-s
> >
> > It's not yet the default and can't be inadvertently enabled so chances
> > are it's the original holes implementation.
>
> The filesystem was definitely created a while ago (2-3 years?)
>
> I have been recently been playing with bees, but I'm reasonably sure I didn't run in on that filesystem
> it lists support for "HOLE extents and btrfs no-holes feature"

It's default except for LZO.


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing'
  2020-05-25 22:47                 ` Chris Murphy
@ 2020-05-25 22:51                   ` Chris Murphy
  2020-05-26  0:13                   ` Marc MERLIN
  1 sibling, 0 replies; 479+ messages in thread
From: Chris Murphy @ 2020-05-25 22:51 UTC (permalink / raw)
  To: Chris Murphy
  Cc: Marc MERLIN, Su Yue, Btrfs BTRFS, Qu Wenruo, Josef Bacik, Su Yue

On Mon, May 25, 2020 at 4:47 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Mon, May 25, 2020 at 2:39 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Mon, May 25, 2020 at 02:24:03PM -0600, Chris Murphy wrote:
> > > OK I didn't understand that the problem is with only the sending file
> > > system, not the receive file system. And also it sounds like the send
> > > did not cause the problem, but it's somehow a pre-existing problem
> > > that --repair isn't completely fixing up, or maybe is making different
> > > (or worse).
> >
> > Correct on all points.
> >
> > > So I guess the real question is what happened to this file system
> > > before the send, that got it into this weird state.
> >
> > That too, but honestly there are a lot of variables, and it feels like a
> > bit of wild goose chase.
>
> Maybe. The story arc on Btrfs is that check --repair only fixes the
> things it knows how to fix. It's gotten better but still has the scary
> warning, and lately has a 10 second delay to really make sure the user
> meant to use it. And regardless of mode, it's slow and just can't
> scale. Neither does "wipe and restore from backup". So the problems of
> inconsistency need to be understood to avoid the problem in the first
> place.
>
>
> >
> > Basically it looked like the issues with the FS are pretty minor (I was
> > able to cp -av the entire data without any file error), but that btrfs
> > check --repair is unable to make it right, which will likely force me to
> > wipe and restore.
> > I know chedk is WIP, and that's why I'm providing feedback :)
>
> What about finding inode 14058737 and deleting it? In all of the
> listed subvolumes? And then unmount and check again?


Before deleting it, can you check if chattr +C is set? Or what kind of
file is this? Because there's an old loophole in older kernels where
chattr +C could be set, but if defragmented while mounted with a
compression option, it would get compressed. And btrfs check complains
about it, but it's not actually a problem. Newer kernels don't do this
but I'm not certain about the send error's relationship with this.

You could try just making a 'cp --reflink=never' copy of the file. And
delete the original (and all of its copies in all subvolumes). Now
'btrfs check'.


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing'
  2020-05-25 22:47                 ` Chris Murphy
  2020-05-25 22:51                   ` Chris Murphy
@ 2020-05-26  0:13                   ` Marc MERLIN
  1 sibling, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2020-05-26  0:13 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Su Yue, Btrfs BTRFS, Qu Wenruo, Josef Bacik, Su Yue

On Mon, May 25, 2020 at 04:47:31PM -0600, Chris Murphy wrote:
> Maybe. The story arc on Btrfs is that check --repair only fixes the
> things it knows how to fix. It's gotten better but still has the scary

that's always been true indeed :)

> warning, and lately has a 10 second delay to really make sure the user
> meant to use it. And regardless of mode, it's slow and just can't
> scale. Neither does "wipe and restore from backup". So the problems of
> inconsistency need to be understood to avoid the problem in the first
> place.

that's ideal, but honestly it's a multi prong approach. I'm stil hoping
that check --repair can improve.
 
> What about finding inode 14058737 and deleting it? In all of the
> listed subvolumes? And then unmount and check again?
 
it's in all the subvolumes, including the btrfs send ones. 
So sure, I can delete my btrfs send snapshots and have to start over,
but really btrfs check should fix this better than rm, even if it makes
the file not what it was. I can always replace it with a good copy
 
On Mon, May 25, 2020 at 04:51:51PM -0600, Chris Murphy wrote:
> Before deleting it, can you check if chattr +C is set? Or what kind of
> file is this? Because there's an old loophole in older kernels where

saruman:~# lsattr /mnt/mnt/root/usr/share/locale/en_GB/LC_MESSAGES/gstreamer-1.0.mo
-------------------- /mnt/mnt/root/usr/share/locale/en_GB/LC_MESSAGES/gstreamer-1.0.mo

> You could try just making a 'cp --reflink=never' copy of the file. And
> delete the original (and all of its copies in all subvolumes). Now
> 'btrfs check'.

I can restore the file from backup without issues, it's more that with
btrfs send, I'm not allowed to delete the inode until it cycles out, and
I can't cycle it out because btrfs send dies when it sees the
inconsistent state.

I mean, yes, I can do this and delete all the snapshots, I was just
hoping to do slighty better as this is close to wiping the filesystem
and starting over (actually it's mostly the same to be honest)

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* 5.6 pretty massive unexplained btrfs corruption:  parent transid verify failed + open_ctree failed
@ 2020-07-07  3:55                 ` Marc MERLIN
  2020-07-07 14:31                   ` Josef Bacik
  2020-07-08  3:44                   ` Zygo Blaxell
  0 siblings, 2 replies; 479+ messages in thread
From: Marc MERLIN @ 2020-07-07  3:55 UTC (permalink / raw)
  To: linux-btrfs

I'd love to know what went wrong so that it doesn't happen again, but let me know if you'd like data off this 
before I wipe it (which I assume is the only way out at this point)
myth:~# btrfs check --mode=lowmem /dev/mapper/crypt_bcache0
Opening filesystem to check...
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
Ignoring transid failure
leaf parent key incorrect 7325633544192
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file system


I did run bees on that filesystem, but I also just did a full btrfs check on it, and it came back clean:

Opening filesystem to check...
Checking filesystem on /dev/mapper/crypt_bcache4
UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
No device size related problem found
[3/7] checking free space cache
cache and super generation don't match, space cache will be invalidated
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 18089211043840 bytes used, no error found
total csum bytes: 17580412652
total tree bytes: 82326192128
total fs tree bytes: 56795086848
total extent tree bytes: 5154258944
btree space waste bytes: 13682108904
file data blocks allocated: 24050542804992


I then moved it to the target machine, started a btrfs send to it, and it failed quickly (due to a mistake
I had an old btrfs binary on that server, but I'm hoping most of the work is done in kernel space and that the user space
btrfs should not corrupt the disk if it's a bit old)

myth:/mnt# uname -r
5.6.5-amd64-preempt-sysrq-20190817

Soon after, the copy failed:
[ 2575.931316] BTRFS info (device dm-0): use zlib compression, level 3
[ 2575.931329] BTRFS info (device dm-0): disk space caching is enabled
[ 2575.931343] BTRFS info (device dm-0): has skinny extents
[ 2577.286749] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 2607.943516] BTRFS info (device dm-0): enabling ssd optimizations
[ 2708.835200] BTRFS warning (device dm-0): block group 13002170433536 has wrong amount of free space
[ 2708.835209] BTRFS warning (device dm-0): failed to load free space cache for block group 13002170433536, rebuilding it now
[ 2740.589580] BTRFS warning (device dm-0): block group 17151175950336 has wrong amount of free space
[ 2740.589593] BTRFS warning (device dm-0): failed to load free space cache for block group 17151175950336, rebuilding it now
[ 2797.204169] perf: interrupt took too long (3146 > 3138), lowering kernel.perf_event_max_sample_rate to 63500
[ 2882.545242] BTRFS info (device dm-0): the free space cache file (26234763345920) is invalid, skip it
[ 3071.631905] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
[ 3071.643430] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
[ 3071.661985] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
[ 3071.661995] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2210: errno=-5 IO failure
[ 3071.661999] BTRFS info (device dm-0): forced readonly
[ 3071.662567] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
[ 3071.663076] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
[ 3071.663083] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2210: errno=-5 IO failure

Ok, maybe there was an IO failure, although none was shown by the kernel:

however, now the FS is mostly dead:
[ 3649.106084] BTRFS info (device dm-0): use zlib compression, level 3
[ 3649.106095] BTRFS info (device dm-0): disk space caching is enabled
[ 3649.106100] BTRFS info (device dm-0): has skinny extents
[ 3650.445828] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 3652.952110] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 3652.959199] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 3652.959208] BTRFS error (device dm-0): failed to read block groups: -5
[ 3653.002227] BTRFS error (device dm-0): open_ctree failed
[ 3876.808183] BTRFS info (device dm-0): use zlib compression, level 3
[ 3876.808192] BTRFS info (device dm-0): disk space caching is enabled
[ 3876.808195] BTRFS info (device dm-0): has skinny extents
[ 3878.140763] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 3880.623113] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 3880.633290] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 3880.633298] BTRFS error (device dm-0): failed to read block groups: -5
[ 3880.669435] BTRFS error (device dm-0): open_ctree failed
[ 4057.606879] BTRFS info (device dm-0): use zlib compression, level 3
[ 4057.606890] BTRFS info (device dm-0): disk space caching is enabled
[ 4057.606894] BTRFS info (device dm-0): has skinny extents
[ 4058.886212] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 4061.501589] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 4061.503790] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 4061.503799] BTRFS error (device dm-0): failed to read block g


myth:/mnt# btrfs-zero-log /dev/mapper/crypt_bcache0
WARNING: this utility is deprecated, please use 'btrfs rescue zero-log'
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
Ignoring transid failure
leaf parent key incorrect 7325633544192
Clearing log on /dev/mapper/crypt_bcache0, previous log_root 0, level 0


myth:/mnt# mount -t btrfs -o recovery,nospace_cache,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
mount: wrong fs type, bad option, bad superblock on /dev/mapper/crypt_bcache0,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

myth:/mnt# dmtail
[ 6665.975324] BTRFS info (device dm-0): unrecognized mount option 'rootflags=recovery'
[ 6665.975357] BTRFS error (device dm-0): open_ctree failed
[ 6686.664202] BTRFS warning (device dm-0): 'recovery' is deprecated, use 'usebackuproot' instead
[ 6686.664213] BTRFS info (device dm-0): trying to use backup root at mount time
[ 6686.664219] BTRFS info (device dm-0): disabling disk space caching
[ 6686.664224] BTRFS info (device dm-0): force clearing of disk cache
[ 6686.664232] BTRFS info (device dm-0): has skinny extents
[ 6687.911926] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 6690.522785] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 6690.529495] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 6690.529504] BTRFS error (device dm-0): failed to read block groups: -5
[ 6690.556227] BTRFS error (device dm-0): open_ctree failed


myth:/mnt# btrfs restore /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
Skipping snapshot win_ro.20200615_02:49:02
Skipping snapshot 0Notmachines_ro.20200626_16:09:30
Skipping snapshot 1Appliances_ro.20200626_17:08:56
Skipping snapshot debian32_ro.20200626_17:11:57
Skipping snapshot debian64_ro.20200626_17:18:15
Skipping snapshot ubuntu_ro.20200626_17:18:44
Skipping snapshot win_ro.20200626_18:39:13
Skipping snapshot 0Notmachines_ro.20200629_00:48:39
Skipping snapshot 1Appliances_ro.20200629_01:11:44
Skipping snapshot debian32_ro.20200629_01:11:50
Skipping snapshot debian64_ro.20200629_01:19:52
Skipping snapshot ubuntu_ro.20200629_01:20:01
Error copying data for /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
myth:/mnt# l /mnt/btrfs_bigbackup/
is missing a lot of stuff

It's a backup server, I can recreate the data, but it will probably take
2 weeks to copy everything again, and I'd like to know what on earth
happened so that I can avoid having this again.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed
  2020-07-07  3:55                 ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN
@ 2020-07-07 14:31                   ` Josef Bacik
  2020-07-07 17:25                     ` Marc MERLIN
  2020-07-08  3:44                   ` Zygo Blaxell
  1 sibling, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2020-07-07 14:31 UTC (permalink / raw)
  To: Marc MERLIN, linux-btrfs

On 7/6/20 11:55 PM, Marc MERLIN wrote:
> I'd love to know what went wrong so that it doesn't happen again, but let me know if you'd like data off this
> before I wipe it (which I assume is the only way out at this point)
> myth:~# btrfs check --mode=lowmem /dev/mapper/crypt_bcache0
> Opening filesystem to check...
> parent transid verify failed on 7325633544192 wanted 359658 found 359661
> parent transid verify failed on 7325633544192 wanted 359658 found 359661
> parent transid verify failed on 7325633544192 wanted 359658 found 359661
> Ignoring transid failure
> leaf parent key incorrect 7325633544192
> ERROR: failed to read block groups: Operation not permitted
> ERROR: cannot open file system
> 
> 
> I did run bees on that filesystem, but I also just did a full btrfs check on it, and it came back clean:
> 
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/crypt_bcache4
> UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d
> [1/7] checking root items
> Fixed 0 roots.
> [2/7] checking extents
> No device size related problem found
> [3/7] checking free space cache
> cache and super generation don't match, space cache will be invalidated
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 18089211043840 bytes used, no error found
> total csum bytes: 17580412652
> total tree bytes: 82326192128
> total fs tree bytes: 56795086848
> total extent tree bytes: 5154258944
> btree space waste bytes: 13682108904
> file data blocks allocated: 24050542804992
> 
> 
> I then moved it to the target machine, started a btrfs send to it, and it failed quickly (due to a mistake
> I had an old btrfs binary on that server, but I'm hoping most of the work is done in kernel space and that the user space
> btrfs should not corrupt the disk if it's a bit old)
> 
> myth:/mnt# uname -r
> 5.6.5-amd64-preempt-sysrq-20190817
> 
> Soon after, the copy failed:
> [ 2575.931316] BTRFS info (device dm-0): use zlib compression, level 3
> [ 2575.931329] BTRFS info (device dm-0): disk space caching is enabled
> [ 2575.931343] BTRFS info (device dm-0): has skinny extents
> [ 2577.286749] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0

You have a corrupt counter here at mount time, does your logs go back far enough 
to see where those came in?  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed
  2020-07-07 14:31                   ` Josef Bacik
@ 2020-07-07 17:25                     ` Marc MERLIN
  0 siblings, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2020-07-07 17:25 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Jul 07, 2020 at 10:31:32AM -0400, Josef Bacik wrote:
> > Soon after, the copy failed:
> > [ 2575.931316] BTRFS info (device dm-0): use zlib compression, level 3
> > [ 2575.931329] BTRFS info (device dm-0): disk space caching is enabled
> > [ 2575.931343] BTRFS info (device dm-0): has skinny extents
> > [ 2577.286749] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
> 
> You have a corrupt counter here at mount time, does your logs go back far
> enough to see where those came in?  Thanks,

I copied all the logs in the original Email.

Here it is all in one swoop:
https://pastebin.com/GURufP8w

The devices going down and back up is normal, my backup script saw the
backup was over (although in that case that's because it failed) and
automatically unmounted the filesystem and turned off the drives.

I'm not sure how my FS got so corrupted, it's been a while since I've
seen damage that bad, especially pretty unexplained like this.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption:  parent transid verify failed + open_ctree failed
  2020-07-07  3:55                 ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN
  2020-07-07 14:31                   ` Josef Bacik
@ 2020-07-08  3:44                   ` Zygo Blaxell
  2020-07-08  4:10                     ` Marc MERLIN
  2020-08-12 22:34                     ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN
  1 sibling, 2 replies; 479+ messages in thread
From: Zygo Blaxell @ 2020-07-08  3:44 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Jul 06, 2020 at 08:55:30PM -0700, Marc MERLIN wrote:
> I'd love to know what went wrong so that it doesn't happen again, but let me know if you'd like data off this 
> before I wipe it (which I assume is the only way out at this point)
> myth:~# btrfs check --mode=lowmem /dev/mapper/crypt_bcache0
> parent transid verify failed on 7325633544192 wanted 359658 found 359661
> parent transid verify failed on 7325633544192 wanted 359658 found 359661
> parent transid verify failed on 7325633544192 wanted 359658 found 359661
> Ignoring transid failure
> leaf parent key incorrect 7325633544192
> ERROR: failed to read block groups: Operation not permitted
> ERROR: cannot open file system
> 
> 
> I did run bees on that filesystem, but I also just did a full btrfs check on it, and it came back clean:
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/crypt_bcache4
> UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d
> [1/7] checking root items
> Fixed 0 roots.
> [2/7] checking extents
> No device size related problem found
> [3/7] checking free space cache
> cache and super generation don't match, space cache will be invalidated
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 18089211043840 bytes used, no error found
> total csum bytes: 17580412652
> total tree bytes: 82326192128
> total fs tree bytes: 56795086848
> total extent tree bytes: 5154258944
> btree space waste bytes: 13682108904
> file data blocks allocated: 24050542804992
> 
> 
> I then moved it to the target machine, started a btrfs send to it, and it failed quickly (due to a mistake
> I had an old btrfs binary on that server, but I'm hoping most of the work is done in kernel space and that the user space
> btrfs should not corrupt the disk if it's a bit old)

btrfs send has historically had bugs but not filesystem-damaging
ones (just relatively harmless kernel crashes and send failures).
btrfs receive is almost entirely userspace--it can't corrupt anything
that can't be corrupted by normal filesystem operations.

> myth:/mnt# uname -r
> 5.6.5-amd64-preempt-sysrq-20190817
> 
> Soon after, the copy failed:
> [ 2575.931316] BTRFS info (device dm-0): use zlib compression, level 3
> [ 2575.931329] BTRFS info (device dm-0): disk space caching is enabled
> [ 2575.931343] BTRFS info (device dm-0): has skinny extents
> [ 2577.286749] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0

This line does indicate an older problem with the filesystem.  It doesn't
tell us whether the corruption happened yesterday or a year ago.
You will need to look at your older kernel logs for that.

> [ 2607.943516] BTRFS info (device dm-0): enabling ssd optimizations
> [ 2708.835200] BTRFS warning (device dm-0): block group 13002170433536 has wrong amount of free space
> [ 2708.835209] BTRFS warning (device dm-0): failed to load free space cache for block group 13002170433536, rebuilding it now
> [ 2740.589580] BTRFS warning (device dm-0): block group 17151175950336 has wrong amount of free space
> [ 2740.589593] BTRFS warning (device dm-0): failed to load free space cache for block group 17151175950336, rebuilding it now
> [ 2797.204169] perf: interrupt took too long (3146 > 3138), lowering kernel.perf_event_max_sample_rate to 63500
> [ 2882.545242] BTRFS info (device dm-0): the free space cache file (26234763345920) is invalid, skip it

Use space_cache=v2, especially on a big filesystem because space_cache=v1
slows down linearly with filesystem size.  There is no need for these
warnings.  They're probably a symptom rather than cause here.

> [ 3071.631905] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
> [ 3071.643430] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
> [ 3071.661985] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
> [ 3071.661995] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2210: errno=-5 IO failure
> [ 3071.661999] BTRFS info (device dm-0): forced readonly
> [ 3071.662567] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
> [ 3071.663076] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
> [ 3071.663083] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2210: errno=-5 IO failure
> 
> Ok, maybe there was an IO failure, although none was shown by the kernel:

The "IO failure" mentioned here is the earlier parent transid verify
failure.  When the verification fails, the caller gets -EIO.

> however, now the FS is mostly dead:
> [ 3649.106084] BTRFS info (device dm-0): use zlib compression, level 3
> [ 3649.106095] BTRFS info (device dm-0): disk space caching is enabled
> [ 3649.106100] BTRFS info (device dm-0): has skinny extents
> [ 3650.445828] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
> [ 3652.952110] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
> [ 3652.959199] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
> [ 3652.959208] BTRFS error (device dm-0): failed to read block groups: -5
> [ 3653.002227] BTRFS error (device dm-0): open_ctree failed
> [ 3876.808183] BTRFS info (device dm-0): use zlib compression, level 3
> [ 3876.808192] BTRFS info (device dm-0): disk space caching is enabled
> [ 3876.808195] BTRFS info (device dm-0): has skinny extents
> [ 3878.140763] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
> [ 3880.623113] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
> [ 3880.633290] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
> [ 3880.633298] BTRFS error (device dm-0): failed to read block groups: -5
> [ 3880.669435] BTRFS error (device dm-0): open_ctree failed
> [ 4057.606879] BTRFS info (device dm-0): use zlib compression, level 3
> [ 4057.606890] BTRFS info (device dm-0): disk space caching is enabled
> [ 4057.606894] BTRFS info (device dm-0): has skinny extents
> [ 4058.886212] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
> [ 4061.501589] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
> [ 4061.503790] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661

The amount of damage here is small, but it looks like you lost a
superblock update or two (btrfs expected an old page and found a new one)
so the root of the filesystem now points at an old tree that has since
been partially overwritten.

Any part of the filesystem on the other side of the missing metadata pages
is no longer accessible without a brute force search of the metadata.
This might take longer than mkfs+restore with the current btrfs check
--repair, especially if you have to run chunk-recover as well.

> myth:/mnt# btrfs-zero-log /dev/mapper/crypt_bcache0
> WARNING: this utility is deprecated, please use 'btrfs rescue zero-log'
> parent transid verify failed on 7325633544192 wanted 359658 found 359661
> parent transid verify failed on 7325633544192 wanted 359658 found 359661
> parent transid verify failed on 7325633544192 wanted 359658 found 359661
> parent transid verify failed on 7325633544192 wanted 359658 found 359661
> Ignoring transid failure
> leaf parent key incorrect 7325633544192
> Clearing log on /dev/mapper/crypt_bcache0, previous log_root 0, level 0

Clearing log tree will have no effect on parent transid verify failure.
It only helps to work around bugs that occur during log tree replay,
which happens at a later stage of mounting the filesystem.

> myth:/mnt# mount -t btrfs -o recovery,nospace_cache,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
> mount: wrong fs type, bad option, bad superblock on /dev/mapper/crypt_bcache0,
>        missing codepage or helper program, or other error
>        In some cases useful info is found in syslog - try
>        dmesg | tail  or so
> 
> myth:/mnt# dmtail
> [ 6665.975324] BTRFS info (device dm-0): unrecognized mount option 'rootflags=recovery'
> [ 6665.975357] BTRFS error (device dm-0): open_ctree failed
> [ 6686.664202] BTRFS warning (device dm-0): 'recovery' is deprecated, use 'usebackuproot' instead
> [ 6686.664213] BTRFS info (device dm-0): trying to use backup root at mount time
> [ 6686.664219] BTRFS info (device dm-0): disabling disk space caching
> [ 6686.664224] BTRFS info (device dm-0): force clearing of disk cache
> [ 6686.664232] BTRFS info (device dm-0): has skinny extents
> [ 6687.911926] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
> [ 6690.522785] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
> [ 6690.529495] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
> [ 6690.529504] BTRFS error (device dm-0): failed to read block groups: -5
> [ 6690.556227] BTRFS error (device dm-0): open_ctree failed
> 
> 
> myth:/mnt# btrfs restore /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
> Skipping snapshot win_ro.20200615_02:49:02
> Skipping snapshot 0Notmachines_ro.20200626_16:09:30
> Skipping snapshot 1Appliances_ro.20200626_17:08:56
> Skipping snapshot debian32_ro.20200626_17:11:57
> Skipping snapshot debian64_ro.20200626_17:18:15
> Skipping snapshot ubuntu_ro.20200626_17:18:44
> Skipping snapshot win_ro.20200626_18:39:13
> Skipping snapshot 0Notmachines_ro.20200629_00:48:39
> Skipping snapshot 1Appliances_ro.20200629_01:11:44
> Skipping snapshot debian32_ro.20200629_01:11:50
> Skipping snapshot debian64_ro.20200629_01:19:52
> Skipping snapshot ubuntu_ro.20200629_01:20:01
> Error copying data for /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
> Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
> Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
> Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
> Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
> Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
> Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
> Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
> Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
> Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
> Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
> Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
> myth:/mnt# l /mnt/btrfs_bigbackup/
> is missing a lot of stuff
> 
> It's a backup server, I can recreate the data, but it will probably take
> 2 weeks to copy everything again, and I'd like to know what on earth
> happened so that I can avoid having this again.

"parent transid verify failed" is btrfs's gentle way of saying "your
devices have busted write ordering, reconfigure or replace them and
try again."

In a later post in this thread, you posted a log showing your drive model
numbers and firmware revisions.  White Label repackages drives made
by other companies(*) under their own part number (singular, they all
use the same part number), but they leave the firmware revision intact,
so we can look up the firmware revision and see that you have two
different WD models in your md6 array from families with known broken
firmware write caching.

sdd is running firmware 0957, also found in circa-2014 WD Green.
The others are running 01.01RA2 firmware that appears in a model family
that includes some broken WD Green and Red models from a few years back
(including the venerable datavore 80.00A80).  I have a few of the WD
branded versions of these drives.  They are unusable with write cache
enabled.  1 in 10 unclean shutdowns lead to filesystem corruption on
btrfs; on ext4, git and postgresql database corruption.  After disabling
write cache, I've used them for years with no problems.

Hopefully your bcache drive is OK, you didn't post any details on that.
bcache on a drive with buggy firmware write caching fails *spectacularly*.

You can work around buggy write cache firmware with a udev rule like
this to disable write cache on all the drives:

        ACTION=="add|change", SUBSYSTEM=="block", DRIVERS=="sd", KERNEL=="sd*[!0-9]", RUN+="/sbin/hdparm -W 0 $devnode"

Note that in your logs, the kernel reports that 'sdd' has write cache
disabled already, maybe due to lack of firmware support or a conservative
default setting.  That makes it probably the only drive in that array
that is working properly.

bcache could be losing its mind too, but although I've heard a lot
of rumors of bcache bugs, I've yet to catch it having a problem that
wasn't directly caused by bad SSD firmware or host configuration.
If the bcache was configured in writeback mode and it was separated from
the backing device for a while then there could be a consistency issue
that would result in something like this, but bcache is pretty good at
preventing that.

In theory, space_cache=v1 might have consistency issues that lead to some
or all of the symptoms above; however, a) btrfs has checks at multiple
points to detect or prevent that, and b) the inconsistency would have
to be caused by a firmware write cache bug or bcache bug anyway.

There are some other questionable things in your setup:  you have a
mdadm-raid5 with no journal device, so if PPL is also not enabled,
and you are running btrfs on top, then this filesystem is vulnerable
to instant destruction by mdadm-raid5 write hole after a disk fails.
bcache in writeback mode with a single cache pool using multiple physical
backing store devices is vulnerable to extra data corruption failures
in the event that the SSD(s) go bad.  I'm guessing that none of the
backing store drives have scterc support, which will complicate error
recovery with SATA bus timeouts and resets as disks fail (though most
of the problems with that are conveniently also prevented by disabling
write cache).  None of these issues caused problems today, though,
and won't cause a problem until disks start to fail.


(*) their product description text says "other companies", but maybe
White Label is just a part of WD, hiding their shame as they dispose of
unsalable inventory in an unsuspecting market.  Don't know, don't care
enough to find out.

> 
> Thanks,
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>  
> Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption:  parent transid verify failed + open_ctree failed
  2020-07-08  3:44                   ` Zygo Blaxell
@ 2020-07-08  4:10                     ` Marc MERLIN
  2020-07-08  5:49                       ` Zygo Blaxell
  2022-03-29 17:18                       ` Marc MERLIN
  2020-08-12 22:34                     ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN
  1 sibling, 2 replies; 479+ messages in thread
From: Marc MERLIN @ 2020-07-08  4:10 UTC (permalink / raw)
  To: Zygo Blaxell; +Cc: linux-btrfs

Hi Zygo, thanks for the very detailled answer.

A few replies:
> Hopefully your bcache drive is OK, you didn't post any details on that.
> bcache on a drive with buggy firmware write caching fails *spectacularly*.

So, I created this as a bcache backing device so that I can elect to add
a caching device later (which you can't do after the fact otherwise).
There was no caching device here, though.

So before we go on, it seems that my FS is damaged beyond easy repair,
that we know why (drives with broken write caching) and that
unfortunately btrfs gets in states where recovery is hard, at best.
If so, I'll just wipe, apply the fixes you recommended and start over.

On Tue, Jul 07, 2020 at 11:44:07PM -0400, Zygo Blaxell wrote:
> > [ 2575.931343] BTRFS info (device dm-0): has skinny extents
> > [ 2577.286749] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
> 
> This line does indicate an older problem with the filesystem.  It doesn't
> tell us whether the corruption happened yesterday or a year ago.
> You will need to look at your older kernel logs for that.
 
Yeah, this could have been long time ago, but remember that I ran a 2
day long btrfs check on it and it came back clean.

> > [ 2882.545242] BTRFS info (device dm-0): the free space cache file (26234763345920) is invalid, skip it
> 
> Use space_cache=v2, especially on a big filesystem because space_cache=v1
> slows down linearly with filesystem size.  There is no need for these
> warnings.  They're probably a symptom rather than cause here.
 
Thanks for that, was not aware of this.

> > Ok, maybe there was an IO failure, although none was shown by the kernel:
> 
> The "IO failure" mentioned here is the earlier parent transid verify
> failure.  When the verification fails, the caller gets -EIO.
 
Got it, so not a de ice IO failure.

> > [ 4058.886212] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
> > [ 4061.501589] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
> > [ 4061.503790] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
> 
> The amount of damage here is small, but it looks like you lost a
> superblock update or two (btrfs expected an old page and found a new one)
> so the root of the filesystem now points at an old tree that has since
> been partially overwritten.
> 
> Any part of the filesystem on the other side of the missing metadata pages
> is no longer accessible without a brute force search of the metadata.
> This might take longer than mkfs+restore with the current btrfs check
> --repair, especially if you have to run chunk-recover as well.
 
unfortunately btrfs check errors early on, so that's not going to work
without manual options which seem not very obvious.

> > myth:/mnt# btrfs-zero-log /dev/mapper/crypt_bcache0
> > WARNING: this utility is deprecated, please use 'btrfs rescue zero-log'
> > parent transid verify failed on 7325633544192 wanted 359658 found 359661
> > parent transid verify failed on 7325633544192 wanted 359658 found 359661
> > parent transid verify failed on 7325633544192 wanted 359658 found 359661
> > parent transid verify failed on 7325633544192 wanted 359658 found 359661
> > Ignoring transid failure
> > leaf parent key incorrect 7325633544192
> > Clearing log on /dev/mapper/crypt_bcache0, previous log_root 0, level 0
> 
> Clearing log tree will have no effect on parent transid verify failure.
> It only helps to work around bugs that occur during log tree replay,
> which happens at a later stage of mounting the filesystem.

Right, I was just a bit desperate and tried it :)

> In a later post in this thread, you posted a log showing your drive model
> numbers and firmware revisions.  White Label repackages drives made
> by other companies(*) under their own part number (singular, they all
> use the same part number), but they leave the firmware revision intact,
> so we can look up the firmware revision and see that you have two
> different WD models in your md6 array from families with known broken
> firmware write caching.
 
Dann, I'm impressed by that analysis, thank you.

> branded versions of these drives.  They are unusable with write cache
> enabled.  1 in 10 unclean shutdowns lead to filesystem corruption on
> btrfs; on ext4, git and postgresql database corruption.  After disabling
> write cache, I've used them for years with no problems.
 
Gotcha, I'm very glad you were able to figure that out. As you said, I
can disable the write cache.
It's a bit sad however that ext4 would have given me something I could
have recovered, while with btrfs, it's so much harder to recover if
you're not a btrfs FS datastructure expert.
 
> There are some other questionable things in your setup:  you have a
> mdadm-raid5 with no journal device, so if PPL is also not enabled,

Sorry, PPL?

> and you are running btrfs on top, then this filesystem is vulnerable
> to instant destruction by mdadm-raid5 write hole after a disk fails.

wait, if a disk fails, at worst I have a stripe that's half written and
hopefully btrfs fails, goes read only and the transaction does not go
through, so nothing happens except loss of the last written data?

I don't have an external journal because this is an external disk array
I can move between machines. Would you suggest I do something else?

> (*) their product description text says "other companies", but maybe
> White Label is just a part of WD, hiding their shame as they dispose of
> unsalable inventory in an unsuspecting market.  Don't know, don't care
> enough to find out.

:)

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption:  parent transid verify failed + open_ctree failed
  2020-07-08  4:10                     ` Marc MERLIN
@ 2020-07-08  5:49                       ` Zygo Blaxell
  2020-07-08 17:41                         ` Marc MERLIN
  2022-03-29 17:18                       ` Marc MERLIN
  1 sibling, 1 reply; 479+ messages in thread
From: Zygo Blaxell @ 2020-07-08  5:49 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Jul 07, 2020 at 09:10:41PM -0700, Marc MERLIN wrote:
> > branded versions of these drives.  They are unusable with write cache
> > enabled.  1 in 10 unclean shutdowns lead to filesystem corruption on
> > btrfs; on ext4, git and postgresql database corruption.  After disabling
> > write cache, I've used them for years with no problems.
>  
> Gotcha, I'm very glad you were able to figure that out. As you said, I
> can disable the write cache.
> It's a bit sad however that ext4 would have given me something I could
> have recovered, while with btrfs, it's so much harder to recover if
> you're not a btrfs FS datastructure expert.

ext2 is pretty much indestructible if you ignore data integrity.
If you want to destroy ext2 metadata, you have to overwrite it one bit
at a time.  The few cases where damage is not proportional to write
size (indirect block lists, directories) are limited to single inodes.
Every metadata item appears in exactly one location that can be computed,
no searching required.  Recovering from arbitrarily awful firmware is
one of the few things ext2 is really good at.

ext4 is significantly more fragile than ext2 if you use the newer features
like extent and flex_bg, which make the metadata mobile like btrfs.
These can prevent e2fsck from being able to recover a large filesystem.

I switched to btrfs back in 2014 after losing a couple of big ext4
filesystems on arrays of WD Greens, _then_ discovered the problem was
the disk firmware not the filesystem.

> > There are some other questionable things in your setup:  you have a
> > mdadm-raid5 with no journal device, so if PPL is also not enabled,
> 
> Sorry, PPL?

Partial Parity Log.  It can be enabled by mdadm --grow.  It's a mdadm
consistency policy, like the journal, but uses reserved metadata space
instead of a separate device.

> > and you are running btrfs on top, then this filesystem is vulnerable
> > to instant destruction by mdadm-raid5 write hole after a disk fails.
> 
> wait, if a disk fails, at worst I have a stripe that's half written and
> hopefully btrfs fails, goes read only and the transaction does not go
> through, so nothing happens except loss of the last written data?

If the array is degraded, and stripe is partially updated, then there is
a crash or power failure, parity will be out of sync with data blocks
in the stripe, so the missing disk's data cannot be generated from parity.

Both old and new data can be damaged by raid5 write hole.  The data
that is damaged is the block on the missing disk that must be computed
using the contents of all other disks.  The damage affects old and new
data the stripe with equal probability, as the data and parity blocks
rotate from one stripe to the next.  Damaged data in an uncommitted tree
(new data) will be ignored if the transaction is not completed, as no
reference to the root of the uncommitted tree will exist after a crash.
Damaged data in a committed tree (old data) is already committed, and if
it's metadata the damage will also break the filesystem.  In other words,
only old data can be damaged by the write hole, because any new damaged
data will be filtered out by the transaction mechanism.

If you have dup metadata in the btrfs then maybe you can recover from
the mirror copy in another stripe.  Hopefully that's not damaged too,
but since both mirrors are updated at roughly the same time on the same
disks, damage to both copies is quite likely.

mdadm PPL or the journal device finishes partial stripe updates after
a crash or power failure, and avoids this failure mode.

> I don't have an external journal because this is an external disk array
> I can move between machines. Would you suggest I do something else?

Enable PPL on mdadm, or use btrfs raid5 data + raid1 metadata (it's
barely usable and some stuff doesn't work properly, but it can run
a backup server, replace a failed disk, and usually self-repair disk
corruption too).

> > (*) their product description text says "other companies", but maybe
> > White Label is just a part of WD, hiding their shame as they dispose of
> > unsalable inventory in an unsuspecting market.  Don't know, don't care
> > enough to find out.
> 
> :)
> 
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>  
> Home page: http://marc.merlins.org/  
> 

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption:  parent transid verify failed + open_ctree failed
  2020-07-08  5:49                       ` Zygo Blaxell
@ 2020-07-08 17:41                         ` Marc MERLIN
  2020-07-08 17:44                           ` Roman Mamedov
  2020-07-08 22:47                           ` Zygo Blaxell
  0 siblings, 2 replies; 479+ messages in thread
From: Marc MERLIN @ 2020-07-08 17:41 UTC (permalink / raw)
  To: Zygo Blaxell; +Cc: linux-btrfs

On Wed, Jul 08, 2020 at 01:49:05AM -0400, Zygo Blaxell wrote:
> > Sorry, PPL?
> 
> Partial Parity Log.  It can be enabled by mdadm --grow.  It's a mdadm
> consistency policy, like the journal, but uses reserved metadata space
> instead of a separate device.
 
looks like it's incompatible with --bitmap which I was already using.
I'm not sure if --grow was possible to convert from one to another, so I
made a new one, and now I have a 4 day wait for the initial scan
(external array with Sata PMP, those are slow unfortunately)

md6 : active raid5 sdx1[5] sdw1[3] sdv1[2] sdu1[1] sdq1[0]
      23441547264 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UUUU_]
      [>....................]  recovery =  0.0% (2920544/5860386816) finish=4835.4min speed=20189K/sec

Could you confirm that what I did here is ok?

gargamel:~# mdadm --create --verbose /dev/md6 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[quvwx]1
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 512K
mdadm: /dev/sdq1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
mdadm: /dev/sdu1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
mdadm: /dev/sdv1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
mdadm: /dev/sdw1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
mdadm: /dev/sdx1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
mdadm: size set to 5860386816K
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md6 started.

gargamel:~# make-bcache -B /dev/md6
UUID:                   eb898c09-debd-4e86-972f-aecdb59670e2
Set UUID:               e14408a3-8e25-414d-ba4a-a8a7a0d7bdc9
version:                1
block_size:             1
data_offset:            16

gargamel:~# cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64 /dev/bcache4

gargamel:~# cryptsetup luksOpen /dev/bcache4 dshelf6
Enter passphrase for /dev/bcache4:

gargamel:~# /sbin/mkfs.btrfs -L DS6 -O extref -m dup /dev/mapper/dshelf6 


> > wait, if a disk fails, at worst I have a stripe that's half written and
> > hopefully btrfs fails, goes read only and the transaction does not go
> > through, so nothing happens except loss of the last written data?
> 
> If the array is degraded, and stripe is partially updated, then there is
> a crash or power failure, parity will be out of sync with data blocks
> in the stripe, so the missing disk's data cannot be generated from parity.
 
I thought --bitmap would know this and know to discard the last blocks
partiailly written. I guess not?

Either way, it seems that losing just a few blocks in the "wrong" place
is enough to lose most of a btrfs FS. That's disappointing, I thought it
was a bit more fault proof than that.

> > I don't have an external journal because this is an external disk array
> > I can move between machines. Would you suggest I do something else?
> 
> Enable PPL on mdadm, or use btrfs raid5 data + raid1 metadata (it's
> barely usable and some stuff doesn't work properly, but it can run
> a backup server, replace a failed disk, and usually self-repair disk
> corruption too).
 
it's been a while since I've used btrfs raid5, it must have improved
since I last did, but I haven't read that it's become production quality
yet :)

Thanks again for your very helpful answers.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption:  parent transid verify failed + open_ctree failed
  2020-07-08 17:41                         ` Marc MERLIN
@ 2020-07-08 17:44                           ` Roman Mamedov
  2020-07-08 22:47                           ` Zygo Blaxell
  1 sibling, 0 replies; 479+ messages in thread
From: Roman Mamedov @ 2020-07-08 17:44 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Zygo Blaxell, linux-btrfs

On Wed, 8 Jul 2020 10:41:10 -0700
Marc MERLIN <marc@merlins.org> wrote:

> On Wed, Jul 08, 2020 at 01:49:05AM -0400, Zygo Blaxell wrote:
> > > Sorry, PPL?
> > 
> > Partial Parity Log.  It can be enabled by mdadm --grow.  It's a mdadm
> > consistency policy, like the journal, but uses reserved metadata space
> > instead of a separate device.
>  
> looks like it's incompatible with --bitmap which I was already using.
> I'm not sure if --grow was possible to convert from one to another

At least you can definitely remove the bitmap with --bitmap=none in the grow
mode. Then in the second --grow can probably add PPL (unfamiliar with that).

-- 
With respect,
Roman

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption:  parent transid verify failed + open_ctree failed
  2020-07-08 17:41                         ` Marc MERLIN
  2020-07-08 17:44                           ` Roman Mamedov
@ 2020-07-08 22:47                           ` Zygo Blaxell
  1 sibling, 0 replies; 479+ messages in thread
From: Zygo Blaxell @ 2020-07-08 22:47 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jul 08, 2020 at 10:41:10AM -0700, Marc MERLIN wrote:
> On Wed, Jul 08, 2020 at 01:49:05AM -0400, Zygo Blaxell wrote:
> > > Sorry, PPL?
> > 
> > Partial Parity Log.  It can be enabled by mdadm --grow.  It's a mdadm
> > consistency policy, like the journal, but uses reserved metadata space
> > instead of a separate device.
>  
> looks like it's incompatible with --bitmap which I was already using.
> I'm not sure if --grow was possible to convert from one to another, so I
> made a new one, and now I have a 4 day wait for the initial scan
> (external array with Sata PMP, those are slow unfortunately)
> 
> md6 : active raid5 sdx1[5] sdw1[3] sdv1[2] sdu1[1] sdq1[0]
>       23441547264 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UUUU_]
>       [>....................]  recovery =  0.0% (2920544/5860386816) finish=4835.4min speed=20189K/sec
> 
> Could you confirm that what I did here is ok?
> 
> gargamel:~# mdadm --create --verbose /dev/md6 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[quvwx]1
> mdadm: layout defaults to left-symmetric
> mdadm: layout defaults to left-symmetric
> mdadm: chunk size defaults to 512K
> mdadm: /dev/sdq1 appears to be part of a raid array:
>        level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
> mdadm: /dev/sdu1 appears to be part of a raid array:
>        level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
> mdadm: /dev/sdv1 appears to be part of a raid array:
>        level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
> mdadm: /dev/sdw1 appears to be part of a raid array:
>        level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
> mdadm: /dev/sdx1 appears to be part of a raid array:
>        level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
> mdadm: size set to 5860386816K
> Continue creating array? y
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md6 started.
> 
> gargamel:~# make-bcache -B /dev/md6
> UUID:                   eb898c09-debd-4e86-972f-aecdb59670e2
> Set UUID:               e14408a3-8e25-414d-ba4a-a8a7a0d7bdc9
> version:                1
> block_size:             1
> data_offset:            16
> 
> gargamel:~# cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64 /dev/bcache4
> 
> gargamel:~# cryptsetup luksOpen /dev/bcache4 dshelf6
> Enter passphrase for /dev/bcache4:
> 
> gargamel:~# /sbin/mkfs.btrfs -L DS6 -O extref -m dup /dev/mapper/dshelf6 

Looks OK.

> > > wait, if a disk fails, at worst I have a stripe that's half written and
> > > hopefully btrfs fails, goes read only and the transaction does not go
> > > through, so nothing happens except loss of the last written data?
> > 
> > If the array is degraded, and stripe is partially updated, then there is
> > a crash or power failure, parity will be out of sync with data blocks
> > in the stripe, so the missing disk's data cannot be generated from parity.
>  
> I thought --bitmap would know this and know to discard the last blocks
> partiailly written. I guess not?
> 
> Either way, it seems that losing just a few blocks in the "wrong" place
> is enough to lose most of a btrfs FS. That's disappointing, I thought it
> was a bit more fault proof than that.

During a write to a stripe, there's a short window of time when data
and parity blocks are out of sync (each disk completes writes at its
own speed).  If the system is interrupted during this time (crash or
power failure), the stripe on disk has data blocks that don't match
the parity block.

Suppose we have 5 disks.  There is always a 1:1 mapping of disks to blocks
in every RAID5 stripe, so let's call the data blocks on each disk A, B,
C, D, and the parity block P, and ignore the other details.

When all the disks are online, before we do a write to D, we have:

	A ^ B ^ C ^ Dold = Pold          (1)

After the first write, we have this on disk:

	A, B, C, Dnew, Pold

After the second write, we have this on disk:

	A, B, C, Dnew, Pnew

	A ^ B ^ C ^ Dnew = Pnew          (2)

If we lose drive C, we can get it back from either the first or last
state:

	A ^ B ^ Cmissing ^ Dold = Pold

so

	Cmissing = Pold ^ A ^ B ^ Dold (3)

Sub (1) into (3):

	Cmissing = (A ^ B ^ C ^ Dold) ^ A ^ B ^ Dold

Sub A ^ A = 0, B ^ B = 0, Dold ^ Dold = 0:

	Cmissing = C

So we recover C correctly.  The same can be done with Pnew and Dnew.

If there is a power failure or crash and the second write doesn't
happen, then we get:

	Cgarbage = Pold ^ A ^ B ^ Dnew    (4)

Sub (1) into (4):

	Cgarbage = (A ^ B ^ C ^ Dold) ^ A ^ B ^ Dnew

Cancel A ^ A = 0, B ^ B = 0:

	Cgarbage = C ^ Dold ^ Dnew

We didn't get C back, we got C xored with some other stuff.

If the writes happen in the other order, we get:

	Cgarbage2 = Pnew ^ A ^ B ^ Dold    (5)

Sub (2) into (5):

	Cgarbage2 = (A ^ B ^ C ^ Dnew) ^ A ^ B ^ Dold

Cancel A ^ A = 0, B ^ B = 0:

	Cgarbage2 = C ^ Dnew ^ Dold

We didn't get C back here either.  With only 2 disks written, we get the
same garbage in both cases.  If there are more than 2 disks written then
we get multiple possible combinations of writes xored together.

If the array is not degraded, then after the crash, mdadm assumes
the data blocks are correct, and recomputes the parity block from the
data blocks during resync (i.e. it updates Pold to Pnew using Dnew).
The filesystem looks only at the data blocks, and all the data blocks
are present, so this works.

The mdadm bitmaps speed this up by indicating which areas of the disk
contain dirty stripes that may require parity recomputation.  Areas that
have not been written don't need to be resynced.

If the array is degraded, some of the data blocks are missing, so we
can't recompute Pnew from whichever of A, B, C, and D blocks are present.
If we have Pold and Dold it's OK, we can correctly compute A, B, or C if
one is missing.  If we have Pnew and Dnew it's OK.  If P is on the missing
disk it's OK, because we have all the data blocks in the stripe intact
so we don't need P.  If D is missing, we will recompute either Dnew or
Dold depending on whether the P block contains Pnew or Pold, but btrfs
transactions can handle both results so it's OK no matter what D contains.
So far so good, all these cases are OK.  PPL ensures we only have these
cases.

If we have Pold and Dnew or Pnew and Dold, then the computed contents
of A, B, and C will be broken when any of those drives are missing.
Since we weren't writing to blocks A, B, or C in this transaction, but
those are the blocks that are corrupted, we may have broken something
btrfs already committed.  btrfs will see broken csums and reject the data.
If A, B, or C are btrfs metadata blocks, the filesystem is damaged.
Either a mirror copy of metadata or a practical fsck is required to repair.

btrfs transactions cover only the specific blocks that were written.
They don't handle cases where a write to block D corrupts data in other
blocks like A, B, or C.  raid5 in degraded mode has cases like that,
so it doesn't work with btrfs.

> > > I don't have an external journal because this is an external disk array
> > > I can move between machines. Would you suggest I do something else?
> > 
> > Enable PPL on mdadm, or use btrfs raid5 data + raid1 metadata (it's
> > barely usable and some stuff doesn't work properly, but it can run
> > a backup server, replace a failed disk, and usually self-repair disk
> > corruption too).
>  
> it's been a while since I've used btrfs raid5, it must have improved
> since I last did, but I haven't read that it's become production quality
> yet :)

Well, don't try to run a live customer orders database on btrfs raid5.
That won't end well.  It can hold a backup copy of the customer orders
database though, provided you follow the restrictions in

	https://lore.kernel.org/linux-btrfs/20200627032414.GX10769@hungrycats.org/

and avoid (or work around) drives with write-caching bugs.

> Thanks again for your very helpful answers.
> 
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>  
> Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption:  parent transid verify failed + open_ctree failed
  2020-07-08  3:44                   ` Zygo Blaxell
  2020-07-08  4:10                     ` Marc MERLIN
@ 2020-08-12 22:34                     ` Marc MERLIN
  2020-08-13  7:39                       ` Roman Mamedov
  2020-08-14  1:43                       ` Zygo Blaxell
  1 sibling, 2 replies; 479+ messages in thread
From: Marc MERLIN @ 2020-08-12 22:34 UTC (permalink / raw)
  To: Zygo Blaxell; +Cc: linux-btrfs

On Tue, Jul 07, 2020 at 11:44:07PM -0400, Zygo Blaxell wrote:
> sdd is running firmware 0957, also found in circa-2014 WD Green.
> The others are running 01.01RA2 firmware that appears in a model family
> that includes some broken WD Green and Red models from a few years back
> (including the venerable datavore 80.00A80).  I have a few of the WD
> branded versions of these drives.  They are unusable with write cache
> enabled.  1 in 10 unclean shutdowns lead to filesystem corruption on
> btrfs; on ext4, git and postgresql database corruption.  After disabling
> write cache, I've used them for years with no problems.
> 
> Hopefully your bcache drive is OK, you didn't post any details on that.
> bcache on a drive with buggy firmware write caching fails *spectacularly*.
> 
> You can work around buggy write cache firmware with a udev rule like
> this to disable write cache on all the drives:
> 
>         ACTION=="add|change", SUBSYSTEM=="block", DRIVERS=="sd", KERNEL=="sd*[!0-9]", RUN+="/sbin/hdparm -W 0 $devnode"
> 
> Note that in your logs, the kernel reports that 'sdd' has write cache
> disabled already, maybe due to lack of firmware support or a conservative
> default setting.  That makes it probably the only drive in that array
> that is working properly.

Hi Zygo, took a while to rebuild the array, but it's back up, thanks for
your tips.

To avoid such pain in the future, is there a way for me to find out if
my other drives have such problems so that I disable write caching on
them now?

WDC WD10EADS-00L 01.0 PQ: 0 ANSI: 6
SAMSUNG HD102UJ  1AA0 PQ: 0 ANSI: 6
ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5


As for the failure you helped me with (thanks):
In hindsight, I think I know what went wrong. I was running bees on the
array to reclaim duplicate space (1.2TB saved, so it was worth it).
bees does lots of metadata operations, so that would explain why the
last operations that didn't get saved/got corrupted caused the fatal
corruption I ended up with.

Because it's an external array, I do stop it to save power and do this:
umount /dev/mapper/crypt_bcache0
sync
dmsetup remove crypt_bcache0
echo 1 > /sys/block/md6/bcache/stop
mdadm --stop /dev/md6
/etc/init.d/smartmontools stop
sleep 5
pdu disk3 off  <= cuts power

Note that I umount, then sync just in case, then a bunch of stuff to
free up block devices, but by then the btrfs FS has been unmounted and
synced.
Yet, I added sleep 5 for good measure before turning the drives off.
Do you think the cache was so broken that even with all the steps I
took, it failed to flush?

Either way, I now do this as per your recommenation:
grep md6 /proc/mdstat | sed "s/.*raid5 //" | tr ' ' '\012' | sed "s/1.*//" | while read d; do /sbin/hdparm -v -W 0 /dev/$d; done

Thanks
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption:  parent transid verify failed + open_ctree failed
  2020-08-12 22:34                     ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN
@ 2020-08-13  7:39                       ` Roman Mamedov
  2020-08-13 15:07                         ` Marc MERLIN
  2020-08-14  1:43                       ` Zygo Blaxell
  1 sibling, 1 reply; 479+ messages in thread
From: Roman Mamedov @ 2020-08-13  7:39 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Zygo Blaxell, linux-btrfs

On Wed, 12 Aug 2020 15:34:33 -0700
Marc MERLIN <marc@merlins.org> wrote:

> Because it's an external array, I do stop it to save power and do this:
> umount /dev/mapper/crypt_bcache0
> sync
> dmsetup remove crypt_bcache0
> echo 1 > /sys/block/md6/bcache/stop
> mdadm --stop /dev/md6
> /etc/init.d/smartmontools stop
> sleep 5
> pdu disk3 off  <= cuts power
> 
> Note that I umount, then sync just in case, then a bunch of stuff to
> free up block devices, but by then the btrfs FS has been unmounted and
> synced.

Or should have been. If it's a copy-paste from a shell script (unless you do
the whole process manually every time), it doesn't appear like you check the
success status of each line before continuing to the next. Either add
"|| exit 1" to each of these that can fail, or just use "#!/bin/bash -e" in
the script's first line.

As is, imagine umount fails ("target busy"), then dmsetup fails ("device in
use"), then mdadm fails to stop the array, but then you cut power to the
entire thing anyways.

-- 
With respect,
Roman

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption:  parent transid verify failed + open_ctree failed
  2020-08-13  7:39                       ` Roman Mamedov
@ 2020-08-13 15:07                         ` Marc MERLIN
  2020-08-14  2:19                           ` Zygo Blaxell
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2020-08-13 15:07 UTC (permalink / raw)
  To: Roman Mamedov; +Cc: Zygo Blaxell, linux-btrfs

On Thu, Aug 13, 2020 at 12:39:46PM +0500, Roman Mamedov wrote:
> Or should have been. If it's a copy-paste from a shell script (unless you do
> the whole process manually every time), it doesn't appear like you check the
> success status of each line before continuing to the next. Either add
> "|| exit 1" to each of these that can fail, or just use "#!/bin/bash -e" in
> the script's first line.
 
Sorry, yeah, it's a partial copy paste of a script, but the script has 
set -e
so if any command fails, the script stops.

> As is, imagine umount fails ("target busy"), then dmsetup fails ("device in
> use"), then mdadm fails to stop the array, but then you cut power to the
> entire thing anyways.

Correct, and set -e makes sure that does not happen.

I have been using this script for years, and it's been fine, but if the
drives have broken write caching and can fail to write data for more
than 5 seconds, then my script will have cut the power on them before
data was committed.
It feels far reaching a conclusion, but given that we already know that
I did get said corruption and I'm pretty darn confident that it happened
after the last time I shut down the array that way, it means that Zygo
is very likely right on the write caching problem.

Which also also why I'd like to check if I need to turn it off on any on
my other drives, since indeed I often bought cheaper versions of drives,
which maybe had firmware bugs.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption:  parent transid verify failed + open_ctree failed
  2020-08-12 22:34                     ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN
  2020-08-13  7:39                       ` Roman Mamedov
@ 2020-08-14  1:43                       ` Zygo Blaxell
  2020-08-15  4:41                         ` Marc MERLIN
  1 sibling, 1 reply; 479+ messages in thread
From: Zygo Blaxell @ 2020-08-14  1:43 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Aug 12, 2020 at 03:34:33PM -0700, Marc MERLIN wrote:
> On Tue, Jul 07, 2020 at 11:44:07PM -0400, Zygo Blaxell wrote:
> > sdd is running firmware 0957, also found in circa-2014 WD Green.
> > The others are running 01.01RA2 firmware that appears in a model family
> > that includes some broken WD Green and Red models from a few years back
> > (including the venerable datavore 80.00A80).  I have a few of the WD
> > branded versions of these drives.  They are unusable with write cache
> > enabled.  1 in 10 unclean shutdowns lead to filesystem corruption on
> > btrfs; on ext4, git and postgresql database corruption.  After disabling
> > write cache, I've used them for years with no problems.
> > 
> > Hopefully your bcache drive is OK, you didn't post any details on that.
> > bcache on a drive with buggy firmware write caching fails *spectacularly*.
> > 
> > You can work around buggy write cache firmware with a udev rule like
> > this to disable write cache on all the drives:
> > 
> >         ACTION=="add|change", SUBSYSTEM=="block", DRIVERS=="sd", KERNEL=="sd*[!0-9]", RUN+="/sbin/hdparm -W 0 $devnode"
> > 
> > Note that in your logs, the kernel reports that 'sdd' has write cache
> > disabled already, maybe due to lack of firmware support or a conservative
> > default setting.  That makes it probably the only drive in that array
> > that is working properly.
> 
> Hi Zygo, took a while to rebuild the array, but it's back up, thanks for
> your tips.
> 
> To avoid such pain in the future, is there a way for me to find out if
> my other drives have such problems so that I disable write caching on
> them now?

> WDC WD10EADS-00L 01.0 PQ: 0 ANSI: 6
> SAMSUNG HD102UJ  1AA0 PQ: 0 ANSI: 6
> ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
> ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
> ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
> ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
> ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
> WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
> WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
> WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
> WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
> WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5

That "0A80" is the last 4 bytes of "80.00A80" which I mentioned above.
They all need write cache turned off.

I'm told that SC61 firmware has a bug fix for SC60's write cache issues;
however, I have some SC60 drives and haven't had write-cache problems
with them.  They're probably fine.  Same for WD 01.01A01--some WD Green
drives are OK.  I have no data for Samsung firmware.

> As for the failure you helped me with (thanks):
> In hindsight, I think I know what went wrong. I was running bees on the
> array to reclaim duplicate space (1.2TB saved, so it was worth it).
> bees does lots of metadata operations, so that would explain why the
> last operations that didn't get saved/got corrupted caused the fatal
> corruption I ended up with.

If the drive lost data writes then you'd have csum failures on data
blocks.  Some of my WD Reds with 0A80 firmware had those.

There's a lot of metadata writes all the time on btrfs, even with
data-heavy workloads.  You only need to lose any two metadata writes
(one in single or raid0 profile) to break the filesystem.

bees typically only spends a single-digit percentage of its time
manipulating metadata.  Most of the bees IO time is waiting for btrfs
to do slow metadata reads while holding a transaction lock.

> Because it's an external array, I do stop it to save power and do this:
> umount /dev/mapper/crypt_bcache0
> sync
> dmsetup remove crypt_bcache0
> echo 1 > /sys/block/md6/bcache/stop
> mdadm --stop /dev/md6
> /etc/init.d/smartmontools stop
> sleep 5
> pdu disk3 off  <= cuts power
> 
> Note that I umount, then sync just in case, then a bunch of stuff to
> free up block devices, but by then the btrfs FS has been unmounted and
> synced.
> 
> Yet, I added sleep 5 for good measure before turning the drives off.
> Do you think the cache was so broken that even with all the steps I
> took, it failed to flush?

Anything after bcache stop is write cache flush theatre.  If the drive's
write cache firmware is broken, then you can put all the flushes you
want, but the flushes don't do anything that wouldn't also be achieved
by just waiting for the disks to be idle for a while.

5 seconds seems short to me.  I'd keep the drive powered for a minute or
two before turning it off.  Ideally I'd set an idle timeout on the drive
(hdparm -S) and wait until the disk spins itself down (hdparm -C).

> Either way, I now do this as per your recommenation:
> grep md6 /proc/mdstat | sed "s/.*raid5 //" | tr ' ' '\012' | sed "s/1.*//" | while read d; do /sbin/hdparm -v -W 0 /dev/$d; done

Note that after a bus reset while the disks are online, the write caching
flag might revert to default.

Write cache corruption doesn't always occur at shutdown.  It can also
happen if there is a UNC sector or even noise on the SATA cables that
triggers a bus reset.  This wouldn't be detected by the host or btrfs--as
far as the storage stack is concerned, the drive ACKed the flush, so
the data's on the disk, and any losses after that point are equivalent
to disk-level corruption.

> Thanks
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>  
> Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption:  parent transid verify failed + open_ctree failed
  2020-08-13 15:07                         ` Marc MERLIN
@ 2020-08-14  2:19                           ` Zygo Blaxell
  0 siblings, 0 replies; 479+ messages in thread
From: Zygo Blaxell @ 2020-08-14  2:19 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Roman Mamedov, linux-btrfs

On Thu, Aug 13, 2020 at 08:07:51AM -0700, Marc MERLIN wrote:
> On Thu, Aug 13, 2020 at 12:39:46PM +0500, Roman Mamedov wrote:
> > Or should have been. If it's a copy-paste from a shell script (unless you do
> > the whole process manually every time), it doesn't appear like you check the
> > success status of each line before continuing to the next. Either add
> > "|| exit 1" to each of these that can fail, or just use "#!/bin/bash -e" in
> > the script's first line.
>  
> Sorry, yeah, it's a partial copy paste of a script, but the script has 
> set -e
> so if any command fails, the script stops.
> 
> > As is, imagine umount fails ("target busy"), then dmsetup fails ("device in
> > use"), then mdadm fails to stop the array, but then you cut power to the
> > entire thing anyways.
> 
> Correct, and set -e makes sure that does not happen.
> 
> I have been using this script for years, and it's been fine, but if the
> drives have broken write caching and can fail to write data for more
> than 5 seconds, then my script will have cut the power on them before
> data was committed.
> It feels far reaching a conclusion, but given that we already know that
> I did get said corruption and I'm pretty darn confident that it happened
> after the last time I shut down the array that way, it means that Zygo
> is very likely right on the write caching problem.
> 
> Which also also why I'd like to check if I need to turn it off on any on
> my other drives, since indeed I often bought cheaper versions of drives,
> which maybe had firmware bugs.

It would be nice if price correlated to firmware quality, but that's
not how disk vendors make drives.  There's a firmware team that produces
firmware for a disk controller SoC family.  It's a lot of expensive work,
so it only happens once unless a major defect is found.  That firmware
ends up in all the drive models built around that SoC family.  So you
end up with White Label, WD Green, and WD Red drive models that all
run the same handful of firmware builds even though there are dozens of
distinct model numbers sold to different market segments (i.e. prices)
over a period of many years.

WD didn't think it was important to avoid SMR in NAS drives--a
requirement that was so obvious that customers ultimately sued them
over it.  WD certainly wouldn't think it was important for their NAS
drives to have more correct firmware than the low-end discount model,
and nobody sued them over it, so they all got shipped with the same bugs.

> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>  
> Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption:  parent transid verify failed + open_ctree failed
  2020-08-14  1:43                       ` Zygo Blaxell
@ 2020-08-15  4:41                         ` Marc MERLIN
  0 siblings, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2020-08-15  4:41 UTC (permalink / raw)
  To: Zygo Blaxell; +Cc: linux-btrfs, Roman Mamedov

On Thu, Aug 13, 2020 at 09:43:59PM -0400, Zygo Blaxell wrote:
> That "0A80" is the last 4 bytes of "80.00A80" which I mentioned above.
> They all need write cache turned off.
 
Cool. Those were different drives, not sure if they had the same bug.

> I'm told that SC61 firmware has a bug fix for SC60's write cache issues;
> however, I have some SC60 drives and haven't had write-cache problems
> with them.  They're probably fine.  Same for WD 01.01A01--some WD Green
> drives are OK.  I have no data for Samsung firmware.
 
So, as incredibly thankful as I am for you having this information, is
that stuff you just google?
Is there a DB of bad firmwares, and does linux maybe have a partial one
to turn off write caching on known bad drives, or am I talking crazy and
in happy happy land? :)

> bees typically only spends a single-digit percentage of its time
> manipulating metadata.  Most of the bees IO time is waiting for btrfs
> to do slow metadata reads while holding a transaction lock.
 
Gotcha.

> Anything after bcache stop is write cache flush theatre.  If the drive's
> write cache firmware is broken, then you can put all the flushes you
> want, but the flushes don't do anything that wouldn't also be achieved
> by just waiting for the disks to be idle for a while.
> 
> 5 seconds seems short to me.  I'd keep the drive powered for a minute or
> two before turning it off.  Ideally I'd set an idle timeout on the drive
> (hdparm -S) and wait until the disk spins itself down (hdparm -C).
 
I see. I didn't realize it could take that long for things to flush if
the firmware isn't doing the job of flushing when it said it did.

> Note that after a bus reset while the disks are online, the write caching
> flag might revert to default.
 
oh boy, thanks for the warning.

> Write cache corruption doesn't always occur at shutdown.  It can also
> happen if there is a UNC sector or even noise on the SATA cables that
> triggers a bus reset.  This wouldn't be detected by the host or btrfs--as
> far as the storage stack is concerned, the drive ACKed the flush, so
> the data's on the disk, and any losses after that point are equivalent
> to disk-level corruption.

I see, that makes sense.

> It would be nice if price correlated to firmware quality, but that's
> not how disk vendors make drives.  There's a firmware team that produces
> firmware for a disk controller SoC family.  It's a lot of expensive work,
> so it only happens once unless a major defect is found.  That firmware
> ends up in all the drive models built around that SoC family.  So you
> end up with White Label, WD Green, and WD Red drive models that all
> run the same handful of firmware builds even though there are dozens of
> distinct model numbers sold to different market segments (i.e. prices)
> over a period of many years.
 
I see, so I guess I don't feel as bad buying the cheaper white label
versions :)

> WD didn't think it was important to avoid SMR in NAS drives--a
> requirement that was so obvious that customers ultimately sued them
> over it.  WD certainly wouldn't think it was important for their NAS
> drives to have more correct firmware than the low-end discount model,
> and nobody sued them over it, so they all got shipped with the same bugs.

Yeah, I saw that, most BS move ever. They definitely deserve the
backlash coming with it.

Thanks for your answers.
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption:  parent transid verify failed + open_ctree failed
  2020-07-08  4:10                     ` Marc MERLIN
  2020-07-08  5:49                       ` Zygo Blaxell
@ 2022-03-29 17:18                       ` Marc MERLIN
  2022-03-30  5:38                         ` Andrei Borzenkov
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-03-29 17:18 UTC (permalink / raw)
  To: Zygo Blaxell, Josef Bacik, linux-btrfs, Chris Murphy, Su Yue, Qu Wenruo
  Cc: Su Yue

Howdy,

This is the followup I was hoping I'd never have to send.

kernel was 5.7 (long uptime, just upgraded to 5.16).

One raid5 drive failed, and as I was replacing it, another one went
offline, but not in a way that the md5 array was taken down.
I shut the the system down, replaced the bad drive, the 2nd drive that
went down wasn't really down, so I broght back the array with a drive
missing.

mdadm --assemble --run --force /dev/md7 /dev/sd[gijk]1
cryptsetup luksOpen /dev/bcache3 dshelf1a
btrfs device scan --all-devices
mount /dev/mapper/dshelf1a /mnt/btrfs_pool1/

BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
BTRFS error (device dm-17): failed to read chunk tree: -5
BTRFS error (device dm-17): open_ctree failed

It's a pretty massive array that will take a *long* time to recover from
backup. If there is a reaosnable way to bring it back up, I'd be
appreciative.

gargamel:~# mount -o ro,recovery /dev/mapper/dshelf1a /mnt/btrfs_pool1/
BTRFS info (device dm-17): flagging fs with big metadata feature
BTRFS warning (device dm-17): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
BTRFS info (device dm-17): trying to use backup root at mount time
BTRFS info (device dm-17): disk space caching is enabled
BTRFS info (device dm-17): has skinny extents
BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
BTRFS error (device dm-17): failed to read chunk tree: -5
BTRFS error (device dm-17): open_ctree failed

Worst case, if it's dead, I'm still happy to try btrfs check lowmem to
see if it has gotten better.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed
  2022-03-29 17:18                       ` Marc MERLIN
@ 2022-03-30  5:38                         ` Andrei Borzenkov
  2022-03-30 14:39                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Andrei Borzenkov @ 2022-03-30  5:38 UTC (permalink / raw)
  To: Marc MERLIN, Zygo Blaxell, Josef Bacik, linux-btrfs,
	Chris Murphy, Su Yue, Qu Wenruo
  Cc: Su Yue

On 29.03.2022 20:18, Marc MERLIN wrote:
> Howdy,
> 
> This is the followup I was hoping I'd never have to send.
> 
> kernel was 5.7 (long uptime, just upgraded to 5.16).
> 
> One raid5 drive failed, and as I was replacing it, another one went
> offline, but not in a way that the md5 array was taken down.
> I shut the the system down, replaced the bad drive, the 2nd drive that
> went down wasn't really down, so I broght back the array with a drive
> missing.
> 

Sorry, which drive? You had one drive that failed and another drive that
went offline. Which drive you replaced? Which drive is missing now? The
third one? The one that failed? The one that went offline?

> mdadm --assemble --run --force /dev/md7 /dev/sd[gijk]1
> cryptsetup luksOpen /dev/bcache3 dshelf1a
> btrfs device scan --all-devices
> mount /dev/mapper/dshelf1a /mnt/btrfs_pool1/
> 
> BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
> BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
> BTRFS error (device dm-17): failed to read chunk tree: -5
> BTRFS error (device dm-17): open_ctree failed
> 
> It's a pretty massive array that will take a *long* time to recover from
> backup. If there is a reaosnable way to bring it back up, I'd be
> appreciative.
> 
> gargamel:~# mount -o ro,recovery /dev/mapper/dshelf1a /mnt/btrfs_pool1/
> BTRFS info (device dm-17): flagging fs with big metadata feature
> BTRFS warning (device dm-17): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
> BTRFS info (device dm-17): trying to use backup root at mount time
> BTRFS info (device dm-17): disk space caching is enabled
> BTRFS info (device dm-17): has skinny extents
> BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
> BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
> BTRFS error (device dm-17): failed to read chunk tree: -5
> BTRFS error (device dm-17): open_ctree failed
> 
> Worst case, if it's dead, I'm still happy to try btrfs check lowmem to
> see if it has gotten better.
> 
> Thanks,
> Marc


^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed
  2022-03-30  5:38                         ` Andrei Borzenkov
@ 2022-03-30 14:39                           ` Marc MERLIN
  2022-03-31 17:19                             ` Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-03-30 14:39 UTC (permalink / raw)
  To: Andrei Borzenkov
  Cc: Zygo Blaxell, Josef Bacik, linux-btrfs, Chris Murphy, Su Yue,
	Qu Wenruo, Su Yue

On Wed, Mar 30, 2022 at 08:38:19AM +0300, Andrei Borzenkov wrote:
> On 29.03.2022 20:18, Marc MERLIN wrote:
> > Howdy,
> > 
> > This is the followup I was hoping I'd never have to send.
> > 
> > kernel was 5.7 (long uptime, just upgraded to 5.16).
> > 
> > One raid5 drive failed, and as I was replacing it, another one went
> > offline, but not in a way that the md5 array was taken down.
> > I shut the the system down, replaced the bad drive, the 2nd drive that
> > went down wasn't really down, so I broght back the array with a drive
> > missing.
> 
> Sorry, which drive? You had one drive that failed and another drive that
> went offline. Which drive you replaced? Which drive is missing now? The
 
The drive that failed entirely is the one I removed since clearly I
couldn't bring the array back up with it.
The one that went offline during shutdown was working again after
reboot, so I brought the array back with one drive missing as shown
below (4 out of 5)

> third one? The one that failed? The one that went offline?

> > mdadm --assemble --run --force /dev/md7 /dev/sd[gijk]1
> > cryptsetup luksOpen /dev/bcache3 dshelf1a
> > btrfs device scan --all-devices
> > mount /dev/mapper/dshelf1a /mnt/btrfs_pool1/
> > 
> > BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > BTRFS error (device dm-17): failed to read chunk tree: -5
> > BTRFS error (device dm-17): open_ctree failed

But clearly the drive that went offline during shutdown, must have
caused some corruption, even if it all it did was just refuse all
writes.
That said, I was somehow hoping that btrfs could unwind the last writes
that failed/were incomplete and get back to a proper state. I'm still
saddened by how fragile btrfs seems compared to ext4 in those cases
(I've had similar issues happen with ext4 in the past, and was always
able to repair the filesystem even if I lost a few files in the process)

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-03-30 14:39                           ` Marc MERLIN
@ 2022-03-31 17:19                             ` Marc MERLIN
  2022-04-03 23:33                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-03-31 17:19 UTC (permalink / raw)
  To: Andrei Borzenkov
  Cc: Zygo Blaxell, Josef Bacik, linux-btrfs, Chris Murphy, Su Yue, Qu Wenruo

I'm going to wait a few more days before I give up and restore from
backup and will consider whether if I should go back to ext4 which for
me has clearly been more resilient of disk misbehaviours like I get
sometimes (of course, nothing will recover from a real double risk
failure in raid5, but over the last 20 years, it's not been uncommon for
me to see more than one drive kicked out of an array due to SCSI or sata
issues, or a drive having a weird shutdown and being able to come back
after a power cycle with all data except the last blocks written to it).

If I go back with btrfs (despite it being non resilient to any problem
described above, btrfs send is great for backups of course), what are
today's best recommended practices?

Kernel will be 5.16. Filesystem will be 24TB and contain mostly bigger
files (100MB to 10GB).

1) mdadm --create /dev/md7 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[abdef]1 --chunk=256 --bitmap=internal
2) echo 0fb96f02-d8da-45ce-aba7-070a1a8420e3 >  /sys/block/bcache64/bcache/attach 
   gargamel:/dev# cat /sys/block/md7/bcache/cache_mode
   [writethrough] writeback writearound none
3) cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64  /dev/bcache64
4) cryptsetup luksOpen /dev/bcache64 dshelf1
5) mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1

Any other btrfs options I should set for format to improve reliability
first and performance second?
I'm told I should use space_cache=v2, is it default now with btrfs-progs 5.10.1-2 ?

Anything else you recommend given that I indeed I have layers in the
middle. I'm actually considering dropping bcache for this filesystem
given that it could be another cause for corruption that btrfs can't
deal with.

Thanks
Marc

> > > mdadm --assemble --run --force /dev/md7 /dev/sd[gijk]1
> > > cryptsetup luksOpen /dev/bcache3 dshelf1a
> > > btrfs device scan --all-devices
> > > mount /dev/mapper/dshelf1a /mnt/btrfs_pool1/
> > > 
> > > BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > > BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > > BTRFS error (device dm-17): failed to read chunk tree: -5
> > > BTRFS error (device dm-17): open_ctree failed
> 
> But clearly the drive that went offline during shutdown, must have
> caused some corruption, even if it all it did was just refuse all
> writes.
> That said, I was somehow hoping that btrfs could unwind the last writes
> that failed/were incomplete and get back to a proper state. I'm still
> saddened by how fragile btrfs seems compared to ext4 in those cases
> (I've had similar issues happen with ext4 in the past, and was always
> able to repair the filesystem even if I lost a few files in the process)

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-03-31 17:19                             ` Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) Marc MERLIN
@ 2022-04-03 23:33                               ` Josef Bacik
  2022-04-04  1:01                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-03 23:33 UTC (permalink / raw)
  To: Marc MERLIN
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy,
	Su Yue, Qu Wenruo

On Thu, Mar 31, 2022 at 10:19:27AM -0700, Marc MERLIN wrote:
> I'm going to wait a few more days before I give up and restore from
> backup and will consider whether if I should go back to ext4 which for
> me has clearly been more resilient of disk misbehaviours like I get
> sometimes (of course, nothing will recover from a real double risk
> failure in raid5, but over the last 20 years, it's not been uncommon for
> me to see more than one drive kicked out of an array due to SCSI or sata
> issues, or a drive having a weird shutdown and being able to come back
> after a power cycle with all data except the last blocks written to it).
> 
> If I go back with btrfs (despite it being non resilient to any problem
> described above, btrfs send is great for backups of course), what are
> today's best recommended practices?
> 
> Kernel will be 5.16. Filesystem will be 24TB and contain mostly bigger
> files (100MB to 10GB).
> 
> 1) mdadm --create /dev/md7 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[abdef]1 --chunk=256 --bitmap=internal
> 2) echo 0fb96f02-d8da-45ce-aba7-070a1a8420e3 >  /sys/block/bcache64/bcache/attach 
>    gargamel:/dev# cat /sys/block/md7/bcache/cache_mode
>    [writethrough] writeback writearound none
> 3) cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64  /dev/bcache64
> 4) cryptsetup luksOpen /dev/bcache64 dshelf1
> 5) mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1
> 
> Any other btrfs options I should set for format to improve reliability
> first and performance second?
> I'm told I should use space_cache=v2, is it default now with btrfs-progs 5.10.1-2 ?
> 
> Anything else you recommend given that I indeed I have layers in the
> middle. I'm actually considering dropping bcache for this filesystem
> given that it could be another cause for corruption that btrfs can't
> deal with.
>

Sorry shit went real wrong this week.  Can you do

btrfs inspect-internal dump-super -f /dev/whatever

This is going to spit out the backup roots for everything.  I was looking at the
code and realized the backup root code doesn't actually work for the chunk root,
since it's special.  Can you then do

btrfs check --chunk-root <bytenr> /dev/whatever

from whatever the newest backup root is, and then keep rolling back until you
find one that works.  If you find one that does work hooray, I can write
something to swap out the chunk bytenr in your super block and we can carry on.

If not we can go with

btrfs rescue chunk-recover /dev/whatever

and see how that works out.  Thanks,

Josef 

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-03 23:33                               ` Josef Bacik
@ 2022-04-04  1:01                                 ` Marc MERLIN
  2022-04-04 15:08                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-04  1:01 UTC (permalink / raw)
  To: Josef Bacik
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy,
	Su Yue, Qu Wenruo

On Sun, Apr 03, 2022 at 07:33:27PM -0400, Josef Bacik wrote:
> Sorry shit went real wrong this week.  Can you do
> 
> btrfs inspect-internal dump-super -f /dev/whatever

Thanks for the answer.
 
superblock: bytenr=65536, device=/dev/mapper/dshelf1a
---------------------------------------------------------
csum_type               0 (crc32c)
csum_size               4
csum                    0xd3d00183 [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    96539b8c-ccc9-47bf-9e6c-29305890941e
metadata_uuid           96539b8c-ccc9-47bf-9e6c-29305890941e
label                   dshelf1
generation              1602089
root                    13577814573056
sys_array_size          129
chunk_root_generation   1600938
root_level              1
chunk_root              21069824
chunk_root_level        1
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             24004156973056
bytes_used              15113376952320
sectorsize              4096
nodesize                16384
leafsize (deprecated)   16384
stripesize              4096
root_dir                6
num_devices             1
compat_flags            0x0
compat_ro_flags         0x0
incompat_flags          0x169
                        ( MIXED_BACKREF |
                          COMPRESS_LZO |
                          BIG_METADATA |
                          EXTENDED_IREF |
                          SKINNY_METADATA )
cache_generation        1602089
uuid_tree_generation    1602089
dev_item.uuid           8d4b0f25-0de9-47a6-a993-bdd301287f30
dev_item.fsid           96539b8c-ccc9-47bf-9e6c-29305890941e [match]
dev_item.type           0
dev_item.total_bytes    24004156973056
dev_item.bytes_used     15178439589888
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0
sys_chunk_array[2048]:   
        item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520)
                length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP
                io_align 65536 io_width 65536 sector_size 4096
                num_stripes 2 sub_stripes 0
                        stripe 0 devid 1 offset 20971520
                        dev_uuid 8d4b0f25-0de9-47a6-a993-bdd301287f30
                        stripe 1 devid 1 offset 29360128
                        dev_uuid 8d4b0f25-0de9-47a6-a993-bdd301287f30
backup_roots[4]:
        backup 0:
                backup_tree_root:       13577814573056  gen: 1602089    level: 1
                backup_chunk_root:      21069824        gen: 1600938    level: 1
                backup_extent_root:     13577824387072  gen: 1602089    level: 2
                backup_fs_root:         13577799401472  gen: 1602089    level: 2
                backup_dev_root:        15645196861440  gen: 1600938    level: 1
                backup_csum_root:       13577868050432  gen: 1602089    level: 3
                backup_total_bytes:     24004156973056
                backup_bytes_used:      15113376952320
                backup_num_devices:     1

        backup 1:
                backup_tree_root:       13577775284224  gen: 1602086    level: 1
                backup_chunk_root:      21069824        gen: 1600938    level: 1
                backup_extent_root:     13577801826304  gen: 1602086    level: 2
                backup_fs_root:         13577770450944  gen: 1602086    level: 2
                backup_dev_root:        15645196861440  gen: 1600938    level: 1
                backup_csum_root:       13577819963392  gen: 1602086    level: 3
                backup_total_bytes:     24004156973056
                backup_bytes_used:      15113376985088
                backup_num_devices:     1

        backup 2:
                backup_tree_root:       13577821667328  gen: 1602087    level: 1
                backup_chunk_root:      21069824        gen: 1600938    level: 1
                backup_extent_root:     13577825026048  gen: 1602087    level: 2
                backup_fs_root:         13577810673664  gen: 1602087    level: 2
                backup_dev_root:        15645196861440  gen: 1600938    level: 1
                backup_csum_root:       13577829089280  gen: 1602087    level: 3
                backup_total_bytes:     24004156973056
                backup_bytes_used:      15113377001472
                backup_num_devices:     1

        backup 3:
                backup_tree_root:       13577779511296  gen: 1602088    level: 1
                backup_chunk_root:      21069824        gen: 1600938    level: 1
                backup_extent_root:     13577802170368  gen: 1602088    level: 2
                backup_fs_root:         13577773039616  gen: 1602088    level: 2
                backup_dev_root:        15645196861440  gen: 1600938    level: 1
                backup_csum_root:       13577819963392  gen: 1602088    level: 3
                backup_total_bytes:     24004156973056
                backup_bytes_used:      15113377017856
                backup_num_devices:     1


> This is going to spit out the backup roots for everything.  I was looking at the
> code and realized the backup root code doesn't actually work for the chunk root,
> since it's special.  Can you then do
> 
> btrfs check --chunk-root <bytenr> /dev/whatever
> from whatever the newest backup root is, and then keep rolling back until you
> find one that works.  If you find one that does work hooray, I can write
> something to swap out the chunk bytenr in your super block and we can carry on.
 
All four seem to be generation 1600938, backup_chunk_root:21069824

gargamel:/var/cache/apt# btrfs check --chunk-root 21069824 /dev/mapper/dshelf1a
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
leaf parent key incorrect 13577821667328
ERROR: could not setup extent tree
ERROR: cannot open file system

> If not we can go with
> btrfs rescue chunk-recover /dev/whatever
> and see how that works out.  Thanks,

gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a
Scanning: 1509736448 in dev0         

I'll let this run, looks like it might take a while.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04  1:01                                 ` Marc MERLIN
@ 2022-04-04 15:08                                   ` Marc MERLIN
  2022-04-04 17:18                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-04 15:08 UTC (permalink / raw)
  To: Josef Bacik
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Sun, Apr 03, 2022 at 06:01:01PM -0700, Marc MERLIN wrote:
> gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a
> Scanning: 1509736448 in dev0         
> 
> I'll let this run, looks like it might take a while.
 
14h later, still going on. I'll update when it's done, it may take a few
days.
gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a
Scanning: 7049296592896 in dev0

Please let me know about the backup chunk roots that all seemed to be
the same and were apparently all pointing to the same corrupted version?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 15:08                                   ` Marc MERLIN
@ 2022-04-04 17:18                                     ` Josef Bacik
  2022-04-04 17:43                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-04 17:18 UTC (permalink / raw)
  To: Marc MERLIN
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Mon, Apr 4, 2022 at 11:09 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, Apr 03, 2022 at 06:01:01PM -0700, Marc MERLIN wrote:
> > gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a
> > Scanning: 1509736448 in dev0
> >
> > I'll let this run, looks like it might take a while.
>
> 14h later, still going on. I'll update when it's done, it may take a few
> days.
> gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a
> Scanning: 7049296592896 in dev0
>
> Please let me know about the backup chunk roots that all seemed to be
> the same and were apparently all pointing to the same corrupted version?

Ok cool apparently we have code that does this already, can you do

btrfs-find-root -o 3 /dev/whatever

and then use the bytenr's it finds like you did with the backup roots

btrfs check --chunk-bytenr <bytenr> /dev/whatever

and see if one of them works.  You can do this without stopping the
chunk recovery if you want to let that keep going.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 17:18                                     ` Josef Bacik
@ 2022-04-04 17:43                                       ` Marc MERLIN
  2022-04-04 17:53                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-04 17:43 UTC (permalink / raw)
  To: Josef Bacik
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Mon, Apr 04, 2022 at 01:18:48PM -0400, Josef Bacik wrote:
> On Mon, Apr 4, 2022 at 11:09 AM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sun, Apr 03, 2022 at 06:01:01PM -0700, Marc MERLIN wrote:
> > > gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a
> > > Scanning: 1509736448 in dev0
> > >
> > > I'll let this run, looks like it might take a while.
> >
> > 14h later, still going on. I'll update when it's done, it may take a few
> > days.
> > gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a
> > Scanning: 7049296592896 in dev0
> >
> > Please let me know about the backup chunk roots that all seemed to be
> > the same and were apparently all pointing to the same corrupted version?
> 
> Ok cool apparently we have code that does this already, can you do
> 
> btrfs-find-root -o 3 /dev/whatever
> and then use the bytenr's it finds like you did with the backup roots
 
gargamel:~# btrfs-find-root -o 3 /dev/mapper/dshelf1a 
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
leaf parent key incorrect 13577821667328
WARNING: could not setup extent tree, skipping it
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
leaf parent key incorrect 13577821667328
Couldn't setup device tree
Superblock thinks the generation is 1600938
Superblock thinks the level is 1
Found tree root at 21069824 gen 1600938 level 1
Well block 21053440(gen: 1822 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1
Well block 21004288(gen: 1692 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1
Well block 20987904(gen: 1443 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1
Well block 20971520(gen: 1293 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1
Well block 21037056(gen: 1170 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1
Well block 21020672(gen: 773 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1

Not a good sign I assume?


> btrfs check --chunk-bytenr <bytenr> /dev/whatever
> 
> and see if one of them works.  You can do this without stopping the
> chunk recovery if you want to let that keep going.  Thanks,
> 
> Josef
> 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 17:43                                       ` Marc MERLIN
@ 2022-04-04 17:53                                         ` Josef Bacik
  2022-04-04 18:10                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-04 17:53 UTC (permalink / raw)
  To: Marc MERLIN
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Mon, Apr 4, 2022 at 1:44 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Apr 04, 2022 at 01:18:48PM -0400, Josef Bacik wrote:
> > On Mon, Apr 4, 2022 at 11:09 AM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sun, Apr 03, 2022 at 06:01:01PM -0700, Marc MERLIN wrote:
> > > > gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a
> > > > Scanning: 1509736448 in dev0
> > > >
> > > > I'll let this run, looks like it might take a while.
> > >
> > > 14h later, still going on. I'll update when it's done, it may take a few
> > > days.
> > > gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a
> > > Scanning: 7049296592896 in dev0
> > >
> > > Please let me know about the backup chunk roots that all seemed to be
> > > the same and were apparently all pointing to the same corrupted version?
> >
> > Ok cool apparently we have code that does this already, can you do
> >
> > btrfs-find-root -o 3 /dev/whatever
> > and then use the bytenr's it finds like you did with the backup roots
>
> gargamel:~# btrfs-find-root -o 3 /dev/mapper/dshelf1a
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> leaf parent key incorrect 13577821667328
> WARNING: could not setup extent tree, skipping it
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> leaf parent key incorrect 13577821667328
> Couldn't setup device tree
> Superblock thinks the generation is 1600938
> Superblock thinks the level is 1
> Found tree root at 21069824 gen 1600938 level 1
> Well block 21053440(gen: 1822 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1
> Well block 21004288(gen: 1692 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1
> Well block 20987904(gen: 1443 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1
> Well block 20971520(gen: 1293 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1
> Well block 21037056(gen: 1170 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1
> Well block 21020672(gen: 773 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1
>
> Not a good sign I assume?
>

Can you do

btrfs check --chunk-bytenr 21053440 /dev/whatever

also I can't seem to find output from a normal btrfs check run, can I
see that as well?  I'm curious if we could just tell the super block
the right transid for this and everything would be ok.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 17:53                                         ` Josef Bacik
@ 2022-04-04 18:10                                           ` Marc MERLIN
  2022-04-04 18:46                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-04 18:10 UTC (permalink / raw)
  To: Josef Bacik
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Mon, Apr 04, 2022 at 01:53:56PM -0400, Josef Bacik wrote:
> Can you do
> 
> btrfs check --chunk-bytenr 21053440 /dev/whatever
 
I assume you meant chunk-root

gargamel:~# btrfs check --chunk-root 21053440 /dev/mapper/dshelf1a
Opening filesystem to check...
No mapping for 13577814573056-13577814589440
Couldn't map the block 13577814573056
Couldn't map the block 13577814573056
bad tree block 13577814573056, bytenr mismatch, want=13577814573056, have=0
Couldn't read tree root
ERROR: cannot open file system


gargamel:~# btrfs check --chunk-root 21069824 /dev/mapper/dshelf1a
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
leaf parent key incorrect 13577821667328
ERROR: could not setup extent tree
ERROR: cannot open file system

> also I can't seem to find output from a normal btrfs check run, can I
> see that as well?  I'm curious if we could just tell the super block
> the right transid for this and everything would be ok.  Thanks,

gargamel:~# btrfs check /dev/mapper/dshelf1a
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
leaf parent key incorrect 13577821667328
ERROR: could not setup extent tree
ERROR: cannot open file system

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 18:10                                           ` Marc MERLIN
@ 2022-04-04 18:46                                             ` Josef Bacik
  2022-04-04 19:04                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-04 18:46 UTC (permalink / raw)
  To: Marc MERLIN
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Mon, Apr 4, 2022 at 2:10 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Apr 04, 2022 at 01:53:56PM -0400, Josef Bacik wrote:
> > Can you do
> >
> > btrfs check --chunk-bytenr 21053440 /dev/whatever
>
> I assume you meant chunk-root
>

I did, my bad.

>
> gargamel:~# btrfs check /dev/mapper/dshelf1a
> Opening filesystem to check...
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> leaf parent key incorrect 13577821667328
> ERROR: could not setup extent tree
> ERROR: cannot open file system
>

Ok now that I've learned how to read, the real problem is the extent
tree is fucked.  The chunk tree isn't happy, but it appears to be sane
enough to proceed, which is good because that could be unfun to
restore.

Good news is the extent tree is easily re-buildable, lets use

btrfs check --init-extent-tree /dev/whatever

and let that baby do its thing.  Hopefully that rebuilds everything
and you'll be good to go.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 18:46                                             ` Josef Bacik
@ 2022-04-04 19:04                                               ` Marc MERLIN
  2022-04-04 19:52                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-04 19:04 UTC (permalink / raw)
  To: Josef Bacik
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Mon, Apr 04, 2022 at 02:46:57PM -0400, Josef Bacik wrote:
> Ok now that I've learned how to read, the real problem is the extent
> tree is fucked.  The chunk tree isn't happy, but it appears to be sane
> enough to proceed, which is good because that could be unfun to
> restore.
> 
> Good news is the extent tree is easily re-buildable, lets use
> 
> btrfs check --init-extent-tree /dev/whatever
> 
> and let that baby do its thing.  Hopefully that rebuilds everything
> and you'll be good to go.  Thanks,

Thanks for the suggestion, sadly it's not that simple:

gargamel:~# btrfs check --init-extent-tree  /dev/mapper/dshelf1a
WARNING:

	Do not use --repair unless you are advised to do so by a developer
	or an experienced user, and then only after having accepted that no
	fsck can successfully repair all types of filesystem corruption. Eg.
	some software or hardware bugs can fatally damage a volume.
	The operation will start in 10 seconds.
	Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
leaf parent key incorrect 13577821667328
WARNING: could not setup extent tree, skipping it
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
leaf parent key incorrect 13577821667328
Couldn't setup device tree
ERROR: cannot open file system

gargamel:~# btrfs --version
btrfs-progs v5.10.1 

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 19:04                                               ` Marc MERLIN
@ 2022-04-04 19:52                                                 ` Josef Bacik
  2022-04-04 20:33                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-04 19:52 UTC (permalink / raw)
  To: Marc MERLIN
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Mon, Apr 4, 2022 at 3:04 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Apr 04, 2022 at 02:46:57PM -0400, Josef Bacik wrote:
> > Ok now that I've learned how to read, the real problem is the extent
> > tree is fucked.  The chunk tree isn't happy, but it appears to be sane
> > enough to proceed, which is good because that could be unfun to
> > restore.
> >
> > Good news is the extent tree is easily re-buildable, lets use
> >
> > btrfs check --init-extent-tree /dev/whatever
> >
> > and let that baby do its thing.  Hopefully that rebuilds everything
> > and you'll be good to go.  Thanks,
>
> Thanks for the suggestion, sadly it's not that simple:
>
> gargamel:~# btrfs check --init-extent-tree  /dev/mapper/dshelf1a
> WARNING:
>
>         Do not use --repair unless you are advised to do so by a developer
>         or an experienced user, and then only after having accepted that no
>         fsck can successfully repair all types of filesystem corruption. Eg.
>         some software or hardware bugs can fatally damage a volume.
>         The operation will start in 10 seconds.
>         Use Ctrl-C to stop it.
> 10 9 8 7 6 5 4 3 2 1
> Starting repair.
> Opening filesystem to check...
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> leaf parent key incorrect 13577821667328
> WARNING: could not setup extent tree, skipping it
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> leaf parent key incorrect 13577821667328
> Couldn't setup device tree
> ERROR: cannot open file system
>
> gargamel:~# btrfs --version
> btrfs-progs v5.10.1
>

Can you build a recent btrfs-progs from git?  We chucked that error
apparently and I can't figure out where it's complaining.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 19:52                                                 ` Josef Bacik
@ 2022-04-04 20:33                                                   ` Marc MERLIN
  2022-04-04 21:04                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-04 20:33 UTC (permalink / raw)
  To: Josef Bacik
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Mon, Apr 04, 2022 at 03:52:15PM -0400, Josef Bacik wrote:
> Can you build a recent btrfs-progs from git?  We chucked that error
> apparently and I can't figure out where it's complaining.  Thanks,

Sure, here's git master

gargamel:/var/local/src/btrfs-progs# ./btrfs --version
btrfs-progs v5.16.2
gargamel:/var/local/src/btrfs-progs# ./btrfs check --init-extent-tree  /dev/mapper/dshelf1a
WARNING:

	Do not use --repair unless you are advised to do so by a developer
	or an experienced user, and then only after having accepted that no
	fsck can successfully repair all types of filesystem corruption. Eg.
	some software or hardware bugs can fatally damage a volume.
	The operation will start in 10 seconds.
	Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
Couldn't setup device tree
ERROR: cannot open file system


Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 20:33                                                   ` Marc MERLIN
@ 2022-04-04 21:04                                                     ` Josef Bacik
  2022-04-04 21:29                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-04 21:04 UTC (permalink / raw)
  To: Marc MERLIN
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Mon, Apr 4, 2022 at 4:33 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Apr 04, 2022 at 03:52:15PM -0400, Josef Bacik wrote:
> > Can you build a recent btrfs-progs from git?  We chucked that error
> > apparently and I can't figure out where it's complaining.  Thanks,
>
> Sure, here's git master
>

Alright we've entered the "Josef throws code at the problem" portion
of the event.

git clone https://github.com/josefbacik/btrfs-progs.git
git checkout for-marc
<build>
re-run fsck

I wonder if the tree root is fucked and that's why it's not finding
the device tree.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 21:04                                                     ` Josef Bacik
@ 2022-04-04 21:29                                                       ` Marc MERLIN
  2022-04-04 21:40                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-04 21:29 UTC (permalink / raw)
  To: Josef Bacik
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Mon, Apr 04, 2022 at 05:04:52PM -0400, Josef Bacik wrote:
> On Mon, Apr 4, 2022 at 4:33 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Mon, Apr 04, 2022 at 03:52:15PM -0400, Josef Bacik wrote:
> > > Can you build a recent btrfs-progs from git?  We chucked that error
> > > apparently and I can't figure out where it's complaining.  Thanks,
> >
> > Sure, here's git master
> >
> 
> Alright we've entered the "Josef throws code at the problem" portion
> of the event.
 
thanks :)

> git clone https://github.com/josefbacik/btrfs-progs.git
> git checkout for-marc
> <build>
> re-run fsck
> 
> I wonder if the tree root is fucked and that's why it's not finding
> the device tree.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# git remote -v
origin	https://github.com/josefbacik/btrfs-progs.git (fetch)
origin	https://github.com/josefbacik/btrfs-progs.git (push)
gargamel:/var/local/src/btrfs-progs-josefbacik# git branch
* for-marc
  master
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs --version
btrfs-progs v5.16.2 
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree  /dev/mapper/dshelf1a
WARNING:

	Do not use --repair unless you are advised to do so by a developer
	or an experienced user, and then only after having accepted that no
	fsck can successfully repair all types of filesystem corruption. Eg.
	some software or hardware bugs can fatally damage a volume.
	The operation will start in 10 seconds.
	Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
Couldn't find the last root for 4
Couldn't setup device tree
ERROR: cannot open file system

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 21:29                                                       ` Marc MERLIN
@ 2022-04-04 21:40                                                         ` Josef Bacik
  2022-04-04 22:09                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-04 21:40 UTC (permalink / raw)
  To: Marc MERLIN
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Mon, Apr 4, 2022 at 5:29 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Apr 04, 2022 at 05:04:52PM -0400, Josef Bacik wrote:
> > On Mon, Apr 4, 2022 at 4:33 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Mon, Apr 04, 2022 at 03:52:15PM -0400, Josef Bacik wrote:
> > > > Can you build a recent btrfs-progs from git?  We chucked that error
> > > > apparently and I can't figure out where it's complaining.  Thanks,
> > >
> > > Sure, here's git master
> > >
> >
> > Alright we've entered the "Josef throws code at the problem" portion
> > of the event.
>
> thanks :)
>
> > git clone https://github.com/josefbacik/btrfs-progs.git
> > git checkout for-marc
> > <build>
> > re-run fsck
> >
> > I wonder if the tree root is fucked and that's why it's not finding
> > the device tree.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# git remote -v
> origin  https://github.com/josefbacik/btrfs-progs.git (fetch)
> origin  https://github.com/josefbacik/btrfs-progs.git (push)
> gargamel:/var/local/src/btrfs-progs-josefbacik# git branch
> * for-marc
>   master
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs --version
> btrfs-progs v5.16.2

Ok cool, lets do

./btrfs check -r 13577779511296 /dev/mapper/dshelf1a
./btrfs check -r 13577821667328 /dev/mapper/dshelf1a
./btrfs check -r 13577775284224 /dev/mapper/dshelf1a

and see what those say, it looks like the tree root is stale and can't
find the root pointers for anything.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 21:40                                                         ` Josef Bacik
@ 2022-04-04 22:09                                                           ` Marc MERLIN
  2022-04-04 22:34                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-04 22:09 UTC (permalink / raw)
  To: Josef Bacik
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Mon, Apr 04, 2022 at 05:40:14PM -0400, Josef Bacik wrote:
> Ok cool, lets do
> 
> ./btrfs check -r 13577779511296 /dev/mapper/dshelf1a
> ./btrfs check -r 13577821667328 /dev/mapper/dshelf1a
> ./btrfs check -r 13577775284224 /dev/mapper/dshelf1a
> 
> and see what those say, it looks like the tree root is stale and can't
> find the root pointers for anything.  Thanks,

Sure thing, the last one is slightly different (found 1602088)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check -r 13577779511296 /dev/mapper/dshelf1a
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
parent transid verify failed on 13577779511296 wanted 1602089 found 1602242
parent transid verify failed on 13577779511296 wanted 1602089 found 1602242
parent transid verify failed on 13577779511296 wanted 1602089 found 1602242
Ignoring transid failure
ERROR: root [1 0] level 0 does not match 1

Couldn't read tree root
ERROR: cannot open file system
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check -r 13577821667328 /dev/mapper/dshelf1a        
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
ERROR: root [1 0] level 0 does not match 1

Couldn't read tree root
ERROR: cannot open file system
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check -r 13577775284224 /dev/mapper/dshelf1a         
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
parent transid verify failed on 13577775284224 wanted 1602089 found 1602088
parent transid verify failed on 13577775284224 wanted 1602089 found 1602242
parent transid verify failed on 13577775284224 wanted 1602089 found 1602242
Ignoring transid failure
ERROR: root [1 0] level 0 does not match 1

Couldn't read tree root
ERROR: cannot open file system

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 22:09                                                           ` Marc MERLIN
@ 2022-04-04 22:34                                                             ` Josef Bacik
  2022-04-04 22:45                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-04 22:34 UTC (permalink / raw)
  To: Marc MERLIN
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Mon, Apr 4, 2022 at 6:09 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Apr 04, 2022 at 05:40:14PM -0400, Josef Bacik wrote:
> > Ok cool, lets do
> >
> > ./btrfs check -r 13577779511296 /dev/mapper/dshelf1a
> > ./btrfs check -r 13577821667328 /dev/mapper/dshelf1a
> > ./btrfs check -r 13577775284224 /dev/mapper/dshelf1a
> >
> > and see what those say, it looks like the tree root is stale and can't
> > find the root pointers for anything.  Thanks,
>
> Sure thing, the last one is slightly different (found 1602088)
>

Alright, lets see how fucked this thing is

./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a

Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 22:34                                                             ` Josef Bacik
@ 2022-04-04 22:45                                                               ` Marc MERLIN
  2022-04-04 22:52                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-04 22:45 UTC (permalink / raw)
  To: Josef Bacik
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Mon, Apr 04, 2022 at 06:34:39PM -0400, Josef Bacik wrote:
> Alright, lets see how fucked this thing is
> 
> ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal
dump-tree -t ROOT /dev/mapper/dshelf1a
btrfs-progs v5.16.2 
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Couldn't read chunk tree
ERROR: unable to open /dev/mapper/dshelf1a

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 22:45                                                               ` Marc MERLIN
@ 2022-04-04 22:52                                                                 ` Josef Bacik
  2022-04-04 23:18                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-04 22:52 UTC (permalink / raw)
  To: Marc MERLIN
  Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo

On Mon, Apr 4, 2022 at 6:45 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Apr 04, 2022 at 06:34:39PM -0400, Josef Bacik wrote:
> > Alright, lets see how fucked this thing is
> >
> > ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal
> dump-tree -t ROOT /dev/mapper/dshelf1a
> btrfs-progs v5.16.2
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> Couldn't read chunk tree
> ERROR: unable to open /dev/mapper/dshelf1a
>

idk what I expected there, you can pull my tree again, rebuild,
re-run, it should work this time.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 22:52                                                                 ` Josef Bacik
@ 2022-04-04 23:18                                                                   ` Marc MERLIN
  2022-04-04 23:24                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-04 23:18 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

[removed Cc to others]

On Mon, Apr 04, 2022 at 06:52:58PM -0400, Josef Bacik wrote:
> On Mon, Apr 4, 2022 at 6:45 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Mon, Apr 04, 2022 at 06:34:39PM -0400, Josef Bacik wrote:
> > > Alright, lets see how fucked this thing is
> > >
> > > ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a
> >
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal
> > dump-tree -t ROOT /dev/mapper/dshelf1a
> > btrfs-progs v5.16.2
> > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > Couldn't read chunk tree
> > ERROR: unable to open /dev/mapper/dshelf1a
> 
> idk what I expected there, you can pull my tree again, rebuild,
> re-run, it should work this time.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a
btrfs-progs v5.16.2 
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
Couldn't find the last root for 4
Couldn't setup device tree
ERROR: unable to open /dev/mapper/dshelf1a

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 23:18                                                                   ` Marc MERLIN
@ 2022-04-04 23:24                                                                     ` Josef Bacik
  2022-04-04 23:42                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-04 23:24 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Apr 4, 2022 at 7:18 PM Marc MERLIN <marc@merlins.org> wrote:
>
> [removed Cc to others]
>
> On Mon, Apr 04, 2022 at 06:52:58PM -0400, Josef Bacik wrote:
> > On Mon, Apr 4, 2022 at 6:45 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Mon, Apr 04, 2022 at 06:34:39PM -0400, Josef Bacik wrote:
> > > > Alright, lets see how fucked this thing is
> > > >
> > > > ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a
> > >
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal
> > > dump-tree -t ROOT /dev/mapper/dshelf1a
> > > btrfs-progs v5.16.2
> > > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > > Couldn't read chunk tree
> > > ERROR: unable to open /dev/mapper/dshelf1a
> >
> > idk what I expected there, you can pull my tree again, rebuild,
> > re-run, it should work this time.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a
> btrfs-progs v5.16.2
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> Couldn't find the last root for 4
> Couldn't setup device tree
> ERROR: unable to open /dev/mapper/dshelf1a
>

Re-pull and try again please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 23:24                                                                     ` Josef Bacik
@ 2022-04-04 23:42                                                                       ` Marc MERLIN
  2022-04-05  0:08                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-04 23:42 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Apr 04, 2022 at 07:24:10PM -0400, Josef Bacik wrote:
> Re-pull and try again please.  Thanks,

Looks the same?

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a
btrfs-progs v5.16.2 
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
Couldn't find the last root for 4
Couldn't setup device tree
ERROR: unable to open /dev/mapper/dshelf1a

gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -12
commit d58725072227f6bd41633cee71695b6bdc17d67f
Author: Josef Bacik <josef@toxicpanda.com>
Date:   Mon Apr 4 19:23:40 2022 -0400

    try harder to open a corrupt fs

commit 1f5227d8de1dbff1b465fdff0177f833ac9df000
Author: Josef Bacik <josef@toxicpanda.com>
Date:   Mon Apr 4 18:52:05 2022 -0400

    allow dump-tree to ignore transid mismatches

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-04 23:42                                                                       ` Marc MERLIN
@ 2022-04-05  0:08                                                                         ` Josef Bacik
  2022-04-05  0:13                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05  0:08 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Apr 4, 2022 at 7:42 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Apr 04, 2022 at 07:24:10PM -0400, Josef Bacik wrote:
> > Re-pull and try again please.  Thanks,
>
> Looks the same?
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a
> btrfs-progs v5.16.2
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> Couldn't find the last root for 4
> Couldn't setup device tree
> ERROR: unable to open /dev/mapper/dshelf1a
>

Fuck it, I'll just comment it out.  Try it again please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  0:08                                                                         ` Josef Bacik
@ 2022-04-05  0:13                                                                           ` Marc MERLIN
  2022-04-05  0:15                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05  0:13 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Apr 04, 2022 at 08:08:53PM -0400, Josef Bacik wrote:
> Fuck it, I'll just comment it out.  Try it again please.  Thanks,

same output :-/

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a
btrfs-progs v5.16.2 
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
Couldn't find the last root for 4
Couldn't setup device tree
ERROR: unable to open /dev/mapper/dshelf1a
gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -12
commit 25024ece18db0a9ef554892098125f2ac5e27f97
Author: Josef Bacik <josef@toxicpanda.com>
Date:   Mon Apr 4 20:08:05 2022 -0400
mel:/var/local/src/btrfs-progs-josefbacik# l btrfs
-rwxr-xr-x 1 root staff 6216888 Apr  4 17:11 btrfs

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  0:13                                                                           ` Marc MERLIN
@ 2022-04-05  0:15                                                                             ` Josef Bacik
  2022-04-05  0:18                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05  0:15 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Apr 4, 2022 at 8:13 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Apr 04, 2022 at 08:08:53PM -0400, Josef Bacik wrote:
> > Fuck it, I'll just comment it out.  Try it again please.  Thanks,
>
> same output :-/
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a
> btrfs-progs v5.16.2
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> Couldn't find the last root for 4
> Couldn't setup device tree
> ERROR: unable to open /dev/mapper/dshelf1a
> gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -12
> commit 25024ece18db0a9ef554892098125f2ac5e27f97
> Author: Josef Bacik <josef@toxicpanda.com>
> Date:   Mon Apr 4 20:08:05 2022 -0400
> mel:/var/local/src/btrfs-progs-josefbacik# l btrfs
> -rwxr-xr-x 1 root staff 6216888 Apr  4 17:11 btrfs
>

Hmm I wonder if the dependencies aren't done right, can you

make clean
make

then rerun?  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  0:15                                                                             ` Josef Bacik
@ 2022-04-05  0:18                                                                               ` Marc MERLIN
  2022-04-05  0:24                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05  0:18 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Apr 04, 2022 at 08:15:45PM -0400, Josef Bacik wrote:
> Hmm I wonder if the dependencies aren't done right, can you
> 
> make clean
> make
> 
sauron [mc]# mv btrfs btrfs.orig
sauron [mc]# make clean; make
(...)
sauron [mc]# diff btrfs btrfs.orig
sauron [mc]# l btrfs btrfs.orig
-rwxr-xr-x 1 root staff 6216888 Apr  4 17:17 btrfs*
-rwxr-xr-x 1 root staff 6216888 Apr  4 17:11 btrfs.orig*

Binary identical after rebuild.
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  0:18                                                                               ` Marc MERLIN
@ 2022-04-05  0:24                                                                                 ` Josef Bacik
  2022-04-05  0:28                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05  0:24 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Apr 4, 2022 at 8:18 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Apr 04, 2022 at 08:15:45PM -0400, Josef Bacik wrote:
> > Hmm I wonder if the dependencies aren't done right, can you
> >
> > make clean
> > make
> >
> sauron [mc]# mv btrfs btrfs.orig
> sauron [mc]# make clean; make
> (...)
> sauron [mc]# diff btrfs btrfs.orig
> sauron [mc]# l btrfs btrfs.orig
> -rwxr-xr-x 1 root staff 6216888 Apr  4 17:17 btrfs*
> -rwxr-xr-x 1 root staff 6216888 Apr  4 17:11 btrfs.orig*
>
> Binary identical after rebuild.

Sigh time for printf sanity checks, thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  0:24                                                                                 ` Josef Bacik
@ 2022-04-05  0:28                                                                                   ` Marc MERLIN
  2022-04-05  0:39                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05  0:28 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Apr 04, 2022 at 08:24:55PM -0400, Josef Bacik wrote:
> > Binary identical after rebuild.
> 
> Sigh time for printf sanity checks, thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -3 
commit 7839d7519ffb3b5d1d102536eb768c25a69c4c05
Author: Josef Bacik <josef@toxicpanda.com>
Date:   Mon Apr 4 20:24:18 2022 -0400
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a
btrfs-progs v5.16.2 
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
FS_INFO IS 0x55ea174c4830
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x55ea174c4830
ERROR: unable to open /dev/mapper/dshelf1a

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  0:28                                                                                   ` Marc MERLIN
@ 2022-04-05  0:39                                                                                     ` Josef Bacik
  2022-04-05  0:58                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05  0:39 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Apr 4, 2022 at 8:28 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Apr 04, 2022 at 08:24:55PM -0400, Josef Bacik wrote:
> > > Binary identical after rebuild.
> >
> > Sigh time for printf sanity checks, thanks,
>

I'm dumb, try again please, thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  0:39                                                                                     ` Josef Bacik
@ 2022-04-05  0:58                                                                                       ` Marc MERLIN
  2022-04-05  1:06                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05  0:58 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Apr 04, 2022 at 08:39:14PM -0400, Josef Bacik wrote:
> On Mon, Apr 4, 2022 at 8:28 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Mon, Apr 04, 2022 at 08:24:55PM -0400, Josef Bacik wrote:
> > > > Binary identical after rebuild.
> > >
> > > Sigh time for printf sanity checks, thanks,
> >
> 
> I'm dumb, try again please, thanks,

progress :)

parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
FS_INFO IS 0x563be446b430
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x563be446b430
btrfs-progs v5.16.2 
root tree
node 13577814573056 level 1 items 322 free space 171 generation 1602089 owner ROOT_TREE
node 13577814573056 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
	key (EXTENT_TREE ROOT_ITEM 0) block 13577821667328 gen 1602089
	key (278 INODE_ITEM 0) block 364978405376 gen 533
	key (306 INODE_ITEM 0) block 13577300590592 gen 1601950
	key (366 EXTENT_DATA 0) block 11822267351040 gen 1591706
	key (428 INODE_ITEM 0) block 11970963865600 gen 1449566
	key (489 EXTENT_DATA 0) block 13576971354112 gen 1601940
	key (541 INODE_ITEM 0) block 364866551808 gen 1453942
	key (602 EXTENT_DATA 0) block 11970666151936 gen 1601669
	key (664 INODE_ITEM 0) block 11651970449408 gen 1592655
	key (725 EXTENT_DATA 0) block 12511619350528 gen 1449892
	key (787 INODE_ITEM 0) block 15645177659392 gen 1551626
	key (848 EXTENT_DATA 0) block 11651970482176 gen 1592655
	key (910 INODE_ITEM 0) block 13577663184896 gen 1602069
	key (971 INODE_ITEM 0) block 15645836460032 gen 1592383
	key (1032 INODE_ITEM 0) block 15645836836864 gen 1592383
	key (1093 EXTENT_DATA 0) block 13577870934016 gen 1602089
	key (1153 INODE_ITEM 0) block 10194600820736 gen 1601442
	key (1214 INODE_ITEM 0) block 15645373874176 gen 1593692
	key (1275 INODE_ITEM 0) block 13577130164224 gen 1589778
	key (1336 EXTENT_DATA 0) block 15645703028736 gen 1589924
	key (1398 INODE_ITEM 0) block 11652218060800 gen 1599077
	key (1459 EXTENT_DATA 0) block 15645720608768 gen 1597377
	key (1521 INODE_ITEM 0) block 15645373956096 gen 1593692
	key (1582 EXTENT_DATA 0) block 11652218290176 gen 1599077
	key (1644 INODE_ITEM 0) block 11652218503168 gen 1599077
	key (1705 EXTENT_DATA 0) block 15645124706304 gen 1600922
	key (1767 INODE_ITEM 0) block 15645067100160 gen 1600014
	key (1828 EXTENT_DATA 0) block 13576966029312 gen 1601940
	key (1890 INODE_ITEM 0) block 782933999616 gen 1599367
	key (1951 EXTENT_DATA 0) block 11822337982464 gen 1599620
	key (2013 INODE_ITEM 0) block 13577131016192 gen 1589778
	key (2074 EXTENT_DATA 0) block 13577010249728 gen 1601941
	key (2136 INODE_ITEM 0) block 10679167303680 gen 1589365
	key (2197 EXTENT_DATA 0) block 13576983691264 gen 1601940
	key (2259 INODE_ITEM 0) block 15645786914816 gen 1601203
	key (2320 EXTENT_DATA 0) block 15645786996736 gen 1601203
	key (2382 INODE_ITEM 0) block 10194603294720 gen 1601442
	key (2444 INODE_ITEM 0) block 15645125050368 gen 1600922
	key (2505 INODE_ITEM 0) block 15645125869568 gen 1600922
	key (2566 EXTENT_DATA 0) block 13576928788480 gen 1601938
	key (2628 INODE_ITEM 0) block 15645126426624 gen 1600922
	key (2689 EXTENT_DATA 0) block 15645459152896 gen 1593949
	key (2751 INODE_ITEM 0) block 11652048273408 gen 1601540
	key (2812 EXTENT_DATA 0) block 13577100820480 gen 1600621
	key (2874 INODE_ITEM 0) block 15646063067136 gen 1590067
	key (2935 EXTENT_DATA 0) block 13577101197312 gen 1600621
	key (2997 INODE_ITEM 0) block 13577103032320 gen 1599331
	key (3058 EXTENT_DATA 0) block 13577101639680 gen 1600621
	key (3120 INODE_ITEM 0) block 13577300656128 gen 1601950
	key (3181 EXTENT_DATA 0) block 13577102114816 gen 1600621
	key (3243 INODE_ITEM 0) block 15645910220800 gen 1600395
	key (3305 INODE_ITEM 0) block 13577104310272 gen 1599331
	key (3366 INODE_ITEM 0) block 15645703143424 gen 1589924
	key (3427 EXTENT_DATA 0) block 13576956723200 gen 1601938
	key (3489 INODE_ITEM 0) block 10194606997504 gen 1601442
	key (3550 EXTENT_DATA 0) block 13577102557184 gen 1600621
	key (3612 INODE_ITEM 0) block 11651868557312 gen 1600466
	key (3673 EXTENT_DATA 0) block 15645099851776 gen 1594356
	key (3735 INODE_ITEM 0) block 15645620404224 gen 1594485
	key (3796 EXTENT_DATA 0) block 15645911203840 gen 1600394
	key (3858 INODE_ITEM 0) block 11971147825152 gen 1599202
	key (3919 EXTENT_DATA 0) block 13576956887040 gen 1601938
	key (3981 INODE_ITEM 0) block 11971147857920 gen 1599202
	key (4042 EXTENT_DATA 0) block 11971147907072 gen 1599202
	key (4104 INODE_ITEM 0) block 11971147923456 gen 1599202
	key (4165 EXTENT_DATA 0) block 15645947117568 gen 1594618
	key (4227 INODE_ITEM 0) block 15645830856704 gen 1594078
	key (4288 EXTENT_DATA 0) block 11971148087296 gen 1599202
	key (4350 INODE_ITEM 0) block 11822017118208 gen 1596840
	key (4411 EXTENT_DATA 0) block 11971148169216 gen 1599202
	key (4473 INODE_ITEM 0) block 13577871097856 gen 1602089
	key (4534 EXTENT_DATA 0) block 15645494771712 gen 1594885
	key (4596 INODE_ITEM 0) block 15646063476736 gen 1594749
	key (4657 EXTENT_DATA 0) block 12512415301632 gen 1601928
	key (4719 INODE_ITEM 0) block 15646063509504 gen 1594749
	key (4780 INODE_ITEM 0) block 15645325754368 gen 1595408
	key (4842 INODE_ITEM 0) block 13577397714944 gen 1601962
	key (4903 EXTENT_DATA 0) block 15645716168704 gen 1600306
	key (4965 INODE_ITEM 0) block 15645668786176 gen 1601187
	key (5026 EXTENT_DATA 0) block 15645827448832 gen 1596971
	key (5088 INODE_ITEM 0) block 12512415940608 gen 1601928
	key (5149 EXTENT_DATA 0) block 15645650518016 gen 1596339
	key (5211 INODE_ITEM 0) block 15645050585088 gen 1597228
	key (5272 EXTENT_DATA 0) block 15645067165696 gen 1600014
	key (5334 INODE_ITEM 0) block 13577220980736 gen 1601949
	key (5395 EXTENT_DATA 0) block 15645133520896 gen 1595328
	key (5457 INODE_ITEM 0) block 13577497264128 gen 1595015
	key (5518 EXTENT_DATA 0) block 10194258903040 gen 1596209
	key (5580 INODE_ITEM 0) block 15645917265920 gen 1595143
	key (5641 EXTENT_DATA 0) block 13577668739072 gen 1602069
	key (5703 INODE_ITEM 0) block 15646063820800 gen 1594749
	key (5764 EXTENT_DATA 0) block 15646063837184 gen 1594749
	key (5826 INODE_ITEM 0) block 13577353576448 gen 1595270
	key (5887 EXTENT_DATA 0) block 15646063853568 gen 1594749
	key (5949 INODE_ITEM 0) block 15646063869952 gen 1594749
	key (6010 EXTENT_DATA 0) block 15645831118848 gen 1594078
	key (6072 INODE_ITEM 0) block 13577353658368 gen 1595270
	key (6133 EXTENT_DATA 0) block 13577497919488 gen 1595015
	key (6195 INODE_ITEM 0) block 13577235742720 gen 1601949
	key (6256 EXTENT_DATA 0) block 10679320313856 gen 1600457
	key (6318 INODE_ITEM 0) block 13577561620480 gen 1596098
	key (6379 EXTENT_DATA 0) block 13577353805824 gen 1595270
	key (6441 INODE_ITEM 0) block 15645132898304 gen 1596112
	key (6502 EXTENT_DATA 0) block 15645650731008 gen 1596339
	key (6564 INODE_ITEM 0) block 15645948575744 gen 1596187
	key (6625 EXTENT_DATA 0) block 15645454974976 gen 1595753
	key (6687 INODE_ITEM 0) block 13577190424576 gen 1597100
	key (6749 INODE_ITEM 0) block 13577221177344 gen 1601949
	key (6810 INODE_ITEM 0) block 15645668818944 gen 1601187
	key (6871 EXTENT_DATA 0) block 12512416022528 gen 1601928
	key (6933 INODE_ITEM 0) block 12512416071680 gen 1601928
	key (6994 EXTENT_DATA 0) block 15645668868096 gen 1601187
	key (7056 INODE_ITEM 0) block 13577221226496 gen 1601949
	key (7117 EXTENT_DATA 0) block 15645548511232 gen 1595772
	key (7179 INODE_ITEM 0) block 12512416268288 gen 1601928
	key (7240 EXTENT_DATA 0) block 13577104080896 gen 1600621
	key (7302 INODE_ITEM 0) block 15645326917632 gen 1595408
	key (7363 EXTENT_DATA 0) block 15645326934016 gen 1595408
	key (7425 INODE_ITEM 0) block 15645831577600 gen 1595539
	key (7486 EXTENT_DATA 0) block 15645831610368 gen 1595539
	key (7548 INODE_ITEM 0) block 13577575661568 gen 1602037
	key (7609 EXTENT_DATA 0) block 15645831725056 gen 1595539
	key (7671 INODE_ITEM 0) block 15645831741440 gen 1595539
	key (7732 EXTENT_DATA 0) block 15645831790592 gen 1595539
	key (7794 INODE_ITEM 0) block 13577104392192 gen 1600621
	key (7855 EXTENT_DATA 0) block 15645831856128 gen 1595539
	key (7917 INODE_ITEM 0) block 15645545578496 gen 1596008
	key (7978 EXTENT_DATA 0) block 15645669097472 gen 1601187
	key (8040 INODE_ITEM 0) block 13577122562048 gen 1601944
	key (8101 EXTENT_DATA 0) block 15645771317248 gen 1595811
	key (8163 INODE_ITEM 0) block 15645146808320 gen 1595947
	key (8224 EXTENT_DATA 0) block 15645830512640 gen 1596971
	key (8286 INODE_ITEM 0) block 15645146841088 gen 1595947
	key (8348 INODE_ITEM 0) block 13577355280384 gen 1595270
	key (8409 INODE_ITEM 0) block 15645410263040 gen 1601053
	key (8470 EXTENT_DATA 0) block 12512416858112 gen 1601928
	key (8532 INODE_ITEM 0) block 13577301131264 gen 1601950
	key (8593 EXTENT_DATA 0) block 15645651288064 gen 1596339
	key (8655 INODE_ITEM 0) block 12511560712192 gen 1600489
	key (8717 INODE_ITEM 0) block 12511560925184 gen 1600489
	key (8778 INODE_ITEM 0) block 15645410279424 gen 1601053
	key (8839 EXTENT_DATA 0) block 13577668902912 gen 1602069
	key (8901 INODE_ITEM 0) block 13577106718720 gen 1600621
	key (8962 EXTENT_DATA 0) block 12511562842112 gen 1600489
	key (9024 INODE_ITEM 0) block 15645410328576 gen 1601053
	key (9085 EXTENT_DATA 0) block 15645669228544 gen 1601187
	key (9147 INODE_ITEM 0) block 13577107308544 gen 1600621
	key (9209 INODE_ITEM 0) block 15645651615744 gen 1596339
	key (9270 INODE_ITEM 0) block 15645651632128 gen 1596339
	key (9331 EXTENT_DATA 0) block 13577122971648 gen 1601944
	key (9393 INODE_ITEM 0) block 15645051731968 gen 1597228
	key (9454 EXTENT_DATA 0) block 12511564906496 gen 1600489
	key (9516 INODE_ITEM 0) block 15645058105344 gen 1597228
	key (9578 INODE_ITEM 0) block 15645061201920 gen 1597228
	key (9639 INODE_ITEM 0) block 12511565053952 gen 1600489
	key (9700 EXTENT_DATA 0) block 15645669359616 gen 1601187
	key (9762 INODE_ITEM 0) block 13577550823424 gen 1602036
	key (9823 EXTENT_DATA 0) block 15645410738176 gen 1601053
	key (9885 INODE_ITEM 0) block 13577107865600 gen 1600621
	key (9947 INODE_ITEM 0) block 13577873506304 gen 1602089
	key (10008 INODE_ITEM 0) block 13577108291584 gen 1600621
	key (10070 INODE_ITEM 0) block 13577108307968 gen 1600621
	key (10131 INODE_ITEM 0) block 12511565660160 gen 1600489
	key (10193 INODE_ITEM 0) block 12511567151104 gen 1600489
	key (10254 INODE_ITEM 0) block 13577397747712 gen 1601962
	key (10315 EXTENT_DATA 0) block 12511570444288 gen 1600489
	key (10377 INODE_ITEM 0) block 12511570493440 gen 1600489
	key (10439 INODE_ITEM 0) block 12511570657280 gen 1600489
	key (10500 INODE_ITEM 0) block 15645746413568 gen 1596616
	key (10561 EXTENT_DATA 0) block 15645669736448 gen 1601187
	key (10623 INODE_ITEM 0) block 15645669867520 gen 1601187
	key (10685 INODE_ITEM 0) block 13577669722112 gen 1602069
	key (10746 INODE_ITEM 0) block 15645317120000 gen 1600752
	key (10807 EXTENT_DATA 0) block 15645128458240 gen 1600922
	key (10869 INODE_ITEM 0) block 15645410836480 gen 1601053
	key (10931 INODE_ITEM 0) block 15645410869248 gen 1601053
	key (10992 INODE_ITEM 0) block 15645669982208 gen 1601187
	key (11086 INODE_ITEM 0) block 15645410934784 gen 1601053
	key (11115 INODE_ITEM 0) block 13577810395136 gen 1602084
	key (11177 INODE_ITEM 0) block 13577849110528 gen 1602088
	key (11232 INODE_ITEM 0) block 13577828564992 gen 1602085
	key (11322 ROOT_ITEM 4223) block 13577837477888 gen 1602086
	key (13004 INODE_ITEM 0) block 137504833536 gen 1601371
	key (14024 INODE_ITEM 0) block 13577810739200 gen 1602084
	key (17642 INODE_ITEM 0) block 46358790144 gen 1601370
	key (22853 INODE_ITEM 0) block 15646046289920 gen 1601329
	key (23051 INODE_ITEM 0) block 12512432324608 gen 1601928
	key (23201 INODE_ITEM 0) block 432390144 gen 1601369
	key (23410 INODE_ITEM 0) block 15645432397824 gen 1600181
	key (24093 INODE_ITEM 0) block 13577314729984 gen 1597100
	key (24359 INODE_ITEM 0) block 12512246775808 gen 1592687
	key (24517 INODE_ITEM 0) block 15645536681984 gen 1596008
	key (25179 INODE_ITEM 0) block 15645545480192 gen 1597267
	key (25524 INODE_ITEM 0) block 15645953802240 gen 1596032
	key (25745 INODE_ITEM 0) block 13577867067392 gen 1602089
	key (27417 INODE_ITEM 0) block 10679248257024 gen 1560751
	key (27478 EXTENT_DATA 0) block 11821933953024 gen 1590261
	key (31836 INODE_ITEM 0) block 15645953900544 gen 1596032
	key (33174 INODE_ITEM 0) block 15645501882368 gen 1564185
	key (33195 INODE_ITEM 0) block 15645501898752 gen 1564185
	key (33256 EXTENT_DATA 0) block 15645501964288 gen 1564185
	key (33314 INODE_ITEM 0) block 13576879210496 gen 1581661
	key (33376 INODE_ITEM 0) block 13576879259648 gen 1581661
	key (36142 INODE_ITEM 0) block 15645954031616 gen 1596032
	key (38085 INODE_ITEM 0) block 15645954080768 gen 1596032
	key (41794 INODE_ITEM 0) block 15645954129920 gen 1596032
	key (47471 INODE_ITEM 0) block 15645954195456 gen 1596032
	key (49726 INODE_ITEM 0) block 7283718029312 gen 1600445
	key (53503 INODE_ITEM 0) block 15645536829440 gen 1596008
	key (54250 INODE_ITEM 0) block 6781551640576 gen 1595865
	key (56850 INODE_ITEM 0) block 11970759065600 gen 1590188
	key (57663 INODE_ITEM 0) block 15645399777280 gen 1600155
	key (61323 INODE_ITEM 0) block 15646041735168 gen 1601329
	key (61922 INODE_ITEM 0) block 13577866543104 gen 1602089
	key (72604 INODE_ITEM 0) block 15645308272640 gen 1589881
	key (75479 INODE_ITEM 0) block 15645106438144 gen 1589878
	key (77348 INODE_ITEM 0) block 15645308370944 gen 1589881
	key (77976 INODE_ITEM 0) block 15645106585600 gen 1589878
	key (78251 INODE_ITEM 0) block 11822462091264 gen 1590186
	key (78355 INODE_ITEM 0) block 15645188128768 gen 1589879
	key (78627 INODE_ITEM 0) block 15645566943232 gen 1559929
	key (78956 INODE_ITEM 0) block 15646041817088 gen 1601329
	key (80698 EXTENT_DATA 0) block 15646041882624 gen 1601329
	key (84137 INODE_ITEM 0) block 15645831938048 gen 1560193
	key (84476 INODE_ITEM 0) block 15646041899008 gen 1601329
	key (87291 EXTENT_DATA 0) block 15645349134336 gen 1589884
	key (90310 INODE_ITEM 0) block 15646041915392 gen 1601329
	key (90750 INODE_ITEM 0) block 15645264510976 gen 1589880
	key (91087 INODE_ITEM 0) block 15645264543744 gen 1589880
	key (91252 INODE_ITEM 0) block 15646041948160 gen 1601329
	key (92792 INODE_ITEM 0) block 15645308403712 gen 1589881
	key (99516 INODE_ITEM 0) block 15645262626816 gen 1600937
	key (111330 INODE_ITEM 0) block 15646041964544 gen 1601329
	key (111397 INODE_ITEM 0) block 15646042046464 gen 1601329
	key (111461 EXTENT_DATA 0) block 15646042062848 gen 1601329
	key (118793 INODE_ITEM 0) block 15646042112000 gen 1601329
	key (118910 INODE_ITEM 0) block 15645997793280 gen 1551263
	key (119431 INODE_ITEM 0) block 15645997891584 gen 1551263
	key (119738 INODE_ITEM 0) block 15646042161152 gen 1601329
	key (119962 INODE_ITEM 0) block 15645661495296 gen 1557907
	key (120299 INODE_ITEM 0) block 12512162103296 gen 1595662
	key (120534 INODE_ITEM 0) block 6781699784704 gen 1581636
	key (120854 INODE_ITEM 0) block 15645598416896 gen 1579668
	key (121029 INODE_ITEM 0) block 15646053974016 gen 1600424
	key (121535 INODE_ITEM 0) block 10194495127552 gen 1600453
	key (125999 INODE_ITEM 0) block 15645262692352 gen 1600937
	key (128782 INODE_ITEM 0) block 11160603590656 gen 1538300
	key (128903 INODE_ITEM 0) block 15645277224960 gen 1553667
	key (129646 INODE_ITEM 0) block 15646079418368 gen 1600430
	key (130451 INODE_ITEM 0) block 15645476159488 gen 1548451
	key (131414 INODE_ITEM 0) block 15645412081664 gen 1548443
	key (131754 INODE_ITEM 0) block 782966816768 gen 1568473
	key (132223 INODE_ITEM 0) block 13577811099648 gen 1602084
	key (132420 INODE_ITEM 0) block 15645772152832 gen 1601202
	key (132530 INODE_ITEM 0) block 8733501063168 gen 1595867
	key (137958 INODE_ITEM 0) block 15645772201984 gen 1601202
	key (157591 EXTENT_DATA 0) block 15645772234752 gen 1601202
	key (159785 ROOT_ITEM 1591389) block 13577238380544 gen 1601950
	key (162267 INODE_ITEM 0) block 15645772251136 gen 1601202
	key (163298 ROOT_ITEM 1597234) block 11160516886528 gen 1601457
	key (164620 ROOT_ITEM 1601059) block 13577821503488 gen 1602089
	key (165390 ROOT_ITEM 1602066) block 13577832382464 gen 1602088
	key (FREE_SPACE UNTYPED 41905291264) block 13577305997312 gen 1601950
	key (FREE_SPACE UNTYPED 304435167232) block 13577236987904 gen 1601949
	key (FREE_SPACE UNTYPED 568038785024) block 13577676505088 gen 1602069
	key (FREE_SPACE UNTYPED 831642402816) block 13577873948672 gen 1602089
	key (FREE_SPACE UNTYPED 1095246020608) block 11821928955904 gen 1599077
	key (FREE_SPACE UNTYPED 1359386509312) block 13576985133056 gen 1601940
	key (FREE_SPACE UNTYPED 1622990127104) block 15645538811904 gen 1600234
	key (FREE_SPACE UNTYPED 1887130615808) block 13577019703296 gen 1601941
	key (FREE_SPACE UNTYPED 2150734233600) block 13576865955840 gen 1601938
	key (FREE_SPACE UNTYPED 2414337851392) block 15645142892544 gen 1600922
	key (FREE_SPACE UNTYPED 2678478340096) block 13577306046464 gen 1601950
	key (FREE_SPACE UNTYPED 2942081957888) block 13577178562560 gen 1600621
	key (FREE_SPACE UNTYPED 3206222446592) block 13576867692544 gen 1601938
	key (FREE_SPACE UNTYPED 3469826064384) block 10194651906048 gen 1601442
	key (FREE_SPACE UNTYPED 3733966553088) block 13576867708928 gen 1601938
	key (FREE_SPACE UNTYPED 3997570170880) block 11971165716480 gen 1599202
	key (FREE_SPACE UNTYPED 4261710659584) block 13577148350464 gen 1601944
	key (FREE_SPACE UNTYPED 4525314277376) block 13577873686528 gen 1602089
	key (FREE_SPACE UNTYPED 4789454766080) block 13577399156736 gen 1601962
	key (FREE_SPACE UNTYPED 5053058383872) block 12512437698560 gen 1601928
	key (FREE_SPACE UNTYPED 5317198872576) block 13577237069824 gen 1601949
	key (FREE_SPACE UNTYPED 5580802490368) block 13577677029376 gen 1602069
	key (FREE_SPACE UNTYPED 5844406108160) block 13577503686656 gen 1595270
	key (FREE_SPACE UNTYPED 6108546596864) block 13577237561344 gen 1601949
	key (FREE_SPACE UNTYPED 6372150214656) block 10679368450048 gen 1600457
	key (FREE_SPACE UNTYPED 6636290703360) block 13577237102592 gen 1601949
	key (FREE_SPACE UNTYPED 6899894321152) block 12512438042624 gen 1601928
	key (FREE_SPACE UNTYPED 7164034809856) block 13577237135360 gen 1601949
	key (FREE_SPACE UNTYPED 7427638427648) block 12512439009280 gen 1601928
	key (FREE_SPACE UNTYPED 7691778916352) block 13577580838912 gen 1602037
	key (FREE_SPACE UNTYPED 7955382534144) block 13577178939392 gen 1600621
	key (FREE_SPACE UNTYPED 8219523022848) block 13577149349888 gen 1601944
	key (FREE_SPACE UNTYPED 8483126640640) block 13577306488832 gen 1601950
	key (FREE_SPACE UNTYPED 8746730258432) block 12512439271424 gen 1601928
	key (FREE_SPACE UNTYPED 9010870747136) block 13577677389824 gen 1602069
	key (FREE_SPACE UNTYPED 9274474364928) block 15645677338624 gen 1601187
	key (FREE_SPACE UNTYPED 9542909820928) block 13577149857792 gen 1601944
	key (FREE_SPACE UNTYPED 9802218471424) block 12511725666304 gen 1600489
	key (FREE_SPACE UNTYPED 10066358960128) block 13577561161728 gen 1602036
	key (FREE_SPACE UNTYPED 10329962577920) block 13577874063360 gen 1602089
	key (FREE_SPACE UNTYPED 10594103066624) block 13577399468032 gen 1601962
	key (FREE_SPACE UNTYPED 10857706684416) block 15645677436928 gen 1601187
	key (FREE_SPACE UNTYPED 11121847173120) block 13577677438976 gen 1602069
	key (FREE_SPACE UNTYPED 11385450790912) block 15645781377024 gen 1601202
	key (FREE_SPACE UNTYPED 11649591279616) block 13577820651520 gen 1602084
	key (FREE_SPACE UNTYPED 11935206604800) block 13577820913664 gen 1602084
	key (FREE_SPACE UNTYPED 12209614749696) block 12512439549952 gen 1601928
	key (FREE_SPACE UNTYPED 12473788792832) block 13577873932288 gen 1602089
	key (FREE_SPACE UNTYPED 12755142705152) block 15645976985600 gen 1596032
	key (FREE_SPACE UNTYPED 13023611715584) block 8239004696576 gen 1600445
	key (FREE_SPACE UNTYPED 13350197002240) block 13577871638528 gen 1602089
	key (FREE_SPACE UNTYPED 13660642607104) block 15646046699520 gen 1601329
	key (FREE_SPACE UNTYPED 13938775293952) block 15646046715904 gen 1601329
	key (FREE_SPACE UNTYPED 14247006306304) block 15646046732288 gen 1601329
	key (FREE_SPACE UNTYPED 14520810471424) block 15646046748672 gen 1601329
	key (FREE_SPACE UNTYPED 14831121858560) block 15646046765056 gen 1601329
	key (FREE_SPACE UNTYPED 15109220990976) block 15645276061696 gen 1600937
	key (FREE_SPACE UNTYPED 15359402835968) block 13577819045888 gen 1602084
	key (FREE_SPACE UNTYPED 15666492997632) block 13577823518720 gen 1602089
	key (FREE_SPACE UNTYPED 15954255806464) block 13577821749248 gen 1602089
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
leaf 13577821667328 items 197 free space 4857 generation 1602242 owner EXTENT_TREE
leaf 13577821667328 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
	item 0 key (8733701406720 METADATA_ITEM 0) itemoff 16250 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 1 key (8733701423104 METADATA_ITEM 0) itemoff 16217 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 2 key (8733701439488 METADATA_ITEM 0) itemoff 16184 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 3 key (8733701455872 METADATA_ITEM 0) itemoff 16151 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 4 key (8733701472256 METADATA_ITEM 0) itemoff 16118 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 5 key (8733701488640 METADATA_ITEM 0) itemoff 16085 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 6 key (8733701505024 METADATA_ITEM 0) itemoff 16052 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 7 key (8733701521408 METADATA_ITEM 0) itemoff 16019 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 8 key (8733701537792 METADATA_ITEM 0) itemoff 15986 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 9 key (8733701554176 METADATA_ITEM 0) itemoff 15953 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 10 key (8733701570560 METADATA_ITEM 0) itemoff 15920 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 11 key (8733701586944 METADATA_ITEM 0) itemoff 15887 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 12 key (8733701603328 METADATA_ITEM 0) itemoff 15854 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 13 key (8733701619712 METADATA_ITEM 0) itemoff 15821 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 14 key (8733701636096 METADATA_ITEM 0) itemoff 15788 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 15 key (8733701652480 METADATA_ITEM 0) itemoff 15755 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 16 key (8733701668864 METADATA_ITEM 0) itemoff 15722 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 17 key (8733701685248 METADATA_ITEM 0) itemoff 15689 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 18 key (8733701701632 METADATA_ITEM 0) itemoff 15656 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 19 key (8733701718016 METADATA_ITEM 0) itemoff 15623 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 20 key (8733701734400 METADATA_ITEM 0) itemoff 15590 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 21 key (8733701750784 METADATA_ITEM 0) itemoff 15557 itemsize 33
		refs 1 gen 1584113 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 22 key (8733701783552 METADATA_ITEM 0) itemoff 15524 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 23 key (8733701799936 METADATA_ITEM 0) itemoff 15491 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 24 key (8733701816320 METADATA_ITEM 0) itemoff 15458 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 25 key (8733701832704 METADATA_ITEM 0) itemoff 15425 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 26 key (8733701865472 METADATA_ITEM 0) itemoff 15392 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 27 key (8733701881856 METADATA_ITEM 0) itemoff 15359 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 28 key (8733701898240 METADATA_ITEM 1) itemoff 15326 itemsize 33
		refs 1 gen 1474318 flags TREE_BLOCK
		tree block skinny level 1
		tree block backref root CSUM_TREE
	item 29 key (8733701914624 METADATA_ITEM 0) itemoff 15293 itemsize 33
		refs 1 gen 1474318 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 30 key (8733701931008 METADATA_ITEM 0) itemoff 15260 itemsize 33
		refs 1 gen 1480353 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 31 key (8733701947392 METADATA_ITEM 0) itemoff 15227 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 32 key (8733701963776 METADATA_ITEM 0) itemoff 15194 itemsize 33
		refs 1 gen 1480353 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 33 key (8733701980160 METADATA_ITEM 0) itemoff 15161 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 34 key (8733701996544 METADATA_ITEM 0) itemoff 15128 itemsize 33
		refs 1 gen 1581638 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 35 key (8733702012928 METADATA_ITEM 0) itemoff 15095 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 36 key (8733702029312 METADATA_ITEM 0) itemoff 15062 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 37 key (8733702045696 METADATA_ITEM 0) itemoff 15029 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 38 key (8733702062080 METADATA_ITEM 0) itemoff 14996 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 39 key (8733702078464 METADATA_ITEM 0) itemoff 14963 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 40 key (8733702094848 METADATA_ITEM 0) itemoff 14930 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 41 key (8733702111232 METADATA_ITEM 0) itemoff 14897 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 42 key (8733702127616 METADATA_ITEM 0) itemoff 14864 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 43 key (8733702144000 METADATA_ITEM 0) itemoff 14831 itemsize 33
		refs 1 gen 1581638 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 44 key (8733702176768 METADATA_ITEM 0) itemoff 14798 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 45 key (8733702193152 METADATA_ITEM 0) itemoff 14765 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 46 key (8733702209536 METADATA_ITEM 0) itemoff 14732 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 47 key (8733702225920 METADATA_ITEM 0) itemoff 14699 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 48 key (8733702242304 METADATA_ITEM 0) itemoff 14666 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 49 key (8733702258688 METADATA_ITEM 0) itemoff 14633 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 50 key (8733702275072 METADATA_ITEM 0) itemoff 14600 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 51 key (8733702291456 METADATA_ITEM 0) itemoff 14567 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 52 key (8733702307840 METADATA_ITEM 0) itemoff 14534 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 53 key (8733702324224 METADATA_ITEM 0) itemoff 14501 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 54 key (8733702340608 METADATA_ITEM 0) itemoff 14468 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 55 key (8733702356992 METADATA_ITEM 0) itemoff 14435 itemsize 33
		refs 1 gen 1360804 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 56 key (8733702389760 METADATA_ITEM 0) itemoff 14402 itemsize 33
		refs 1 gen 1360804 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 57 key (8733702406144 METADATA_ITEM 0) itemoff 14369 itemsize 33
		refs 1 gen 1360804 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 58 key (8733702422528 METADATA_ITEM 0) itemoff 14336 itemsize 33
		refs 1 gen 838551 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 59 key (8733702438912 METADATA_ITEM 0) itemoff 14303 itemsize 33
		refs 1 gen 1360804 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 60 key (8733702455296 METADATA_ITEM 0) itemoff 14270 itemsize 33
		refs 1 gen 1360804 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 61 key (8733702504448 METADATA_ITEM 0) itemoff 14237 itemsize 33
		refs 1 gen 300521 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 62 key (8733702520832 METADATA_ITEM 0) itemoff 14204 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 63 key (8733702537216 METADATA_ITEM 0) itemoff 14171 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 64 key (8733702553600 METADATA_ITEM 0) itemoff 14138 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 65 key (8733702569984 METADATA_ITEM 0) itemoff 14105 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 66 key (8733702586368 METADATA_ITEM 0) itemoff 14072 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 67 key (8733702602752 METADATA_ITEM 0) itemoff 14039 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 68 key (8733702619136 METADATA_ITEM 0) itemoff 14006 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 69 key (8733702635520 METADATA_ITEM 0) itemoff 13973 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 70 key (8733702651904 METADATA_ITEM 0) itemoff 13940 itemsize 33
		refs 1 gen 1602171 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 71 key (8733702668288 METADATA_ITEM 0) itemoff 13907 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 72 key (8733702684672 METADATA_ITEM 0) itemoff 13874 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 73 key (8733702701056 METADATA_ITEM 0) itemoff 13841 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 74 key (8733702717440 METADATA_ITEM 0) itemoff 13808 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 75 key (8733702733824 METADATA_ITEM 0) itemoff 13775 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 76 key (8733702750208 METADATA_ITEM 0) itemoff 13742 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 77 key (8733702766592 METADATA_ITEM 0) itemoff 13709 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 78 key (8733702782976 METADATA_ITEM 0) itemoff 13676 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 79 key (8733702799360 METADATA_ITEM 0) itemoff 13643 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 80 key (8733702815744 METADATA_ITEM 0) itemoff 13610 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 81 key (8733702832128 METADATA_ITEM 0) itemoff 13577 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 82 key (8733702848512 METADATA_ITEM 0) itemoff 13544 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 83 key (8733702864896 METADATA_ITEM 0) itemoff 13511 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 84 key (8733702881280 METADATA_ITEM 0) itemoff 13478 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 85 key (8733702897664 METADATA_ITEM 0) itemoff 13445 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 86 key (8733702914048 METADATA_ITEM 0) itemoff 13412 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 87 key (8733702930432 METADATA_ITEM 0) itemoff 13379 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 88 key (8733702946816 METADATA_ITEM 0) itemoff 13346 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 89 key (8733702963200 METADATA_ITEM 0) itemoff 13313 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 90 key (8733702979584 METADATA_ITEM 0) itemoff 13280 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 91 key (8733702995968 METADATA_ITEM 0) itemoff 13247 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 92 key (8733703012352 METADATA_ITEM 0) itemoff 13214 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 93 key (8733703028736 METADATA_ITEM 0) itemoff 13181 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 94 key (8733703045120 METADATA_ITEM 0) itemoff 13148 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 95 key (8733703061504 METADATA_ITEM 0) itemoff 13115 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 96 key (8733703077888 METADATA_ITEM 0) itemoff 13082 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 97 key (8733703094272 METADATA_ITEM 0) itemoff 13049 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 98 key (8733703110656 METADATA_ITEM 0) itemoff 13016 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 99 key (8733703127040 METADATA_ITEM 0) itemoff 12983 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 100 key (8733703143424 METADATA_ITEM 0) itemoff 12950 itemsize 33
		refs 1 gen 4166 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 101 key (8733703159808 METADATA_ITEM 0) itemoff 12917 itemsize 33
		refs 1 gen 1581638 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 102 key (8733703176192 METADATA_ITEM 0) itemoff 12884 itemsize 33
		refs 1 gen 1581638 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 103 key (8733703192576 METADATA_ITEM 0) itemoff 12851 itemsize 33
		refs 1 gen 1514029 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 104 key (8733703208960 METADATA_ITEM 0) itemoff 12818 itemsize 33
		refs 1 gen 4166 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 105 key (8733703225344 METADATA_ITEM 0) itemoff 12785 itemsize 33
		refs 1 gen 1514029 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 106 key (8733703241728 METADATA_ITEM 0) itemoff 12752 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 107 key (8733703258112 METADATA_ITEM 0) itemoff 12719 itemsize 33
		refs 1 gen 1581638 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 108 key (8733703274496 METADATA_ITEM 0) itemoff 12686 itemsize 33
		refs 1 gen 300521 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 109 key (8733703290880 METADATA_ITEM 0) itemoff 12653 itemsize 33
		refs 1 gen 1584112 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 110 key (8733703323648 METADATA_ITEM 0) itemoff 12620 itemsize 33
		refs 1 gen 1602171 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 111 key (8733703340032 METADATA_ITEM 0) itemoff 12587 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 112 key (8733703454720 METADATA_ITEM 0) itemoff 12554 itemsize 33
		refs 1 gen 1535773 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 113 key (8733703471104 METADATA_ITEM 0) itemoff 12521 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 114 key (8733703487488 METADATA_ITEM 0) itemoff 12488 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 115 key (8733703503872 METADATA_ITEM 0) itemoff 12455 itemsize 33
		refs 1 gen 519869 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 116 key (8733703520256 METADATA_ITEM 0) itemoff 12422 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 117 key (8733703536640 METADATA_ITEM 0) itemoff 12389 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 118 key (8733703553024 METADATA_ITEM 0) itemoff 12356 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 119 key (8733703569408 METADATA_ITEM 0) itemoff 12323 itemsize 33
		refs 1 gen 1602171 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 120 key (8733703602176 METADATA_ITEM 0) itemoff 12290 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 121 key (8733703651328 METADATA_ITEM 0) itemoff 12257 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 122 key (8733703733248 METADATA_ITEM 0) itemoff 12224 itemsize 33
		refs 1 gen 1510262 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 123 key (8733703749632 METADATA_ITEM 0) itemoff 12191 itemsize 33
		refs 1 gen 1601423 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 124 key (8733703766016 METADATA_ITEM 0) itemoff 12158 itemsize 33
		refs 1 gen 1509558 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 125 key (8733703798784 METADATA_ITEM 0) itemoff 12125 itemsize 33
		refs 1 gen 1480353 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 126 key (8733703815168 METADATA_ITEM 0) itemoff 12092 itemsize 33
		refs 1 gen 1480353 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 127 key (8733703831552 METADATA_ITEM 0) itemoff 12059 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 128 key (8733703847936 METADATA_ITEM 0) itemoff 12026 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 129 key (8733703864320 METADATA_ITEM 0) itemoff 11993 itemsize 33
		refs 1 gen 3246 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 130 key (8733703880704 METADATA_ITEM 0) itemoff 11960 itemsize 33
		refs 1 gen 3246 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 131 key (8733703897088 METADATA_ITEM 0) itemoff 11927 itemsize 33
		refs 1 gen 1518676 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 132 key (8733703913472 METADATA_ITEM 0) itemoff 11894 itemsize 33
		refs 1 gen 3246 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 133 key (8733703929856 METADATA_ITEM 0) itemoff 11861 itemsize 33
		refs 1 gen 3246 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 134 key (8733703946240 METADATA_ITEM 0) itemoff 11828 itemsize 33
		refs 1 gen 1590258 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root 11223
	item 135 key (8733703979008 METADATA_ITEM 0) itemoff 11795 itemsize 33
		refs 1 gen 749275 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 136 key (8733704060928 METADATA_ITEM 0) itemoff 11762 itemsize 33
		refs 1 gen 1509163 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 137 key (8733704126464 METADATA_ITEM 0) itemoff 11729 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 138 key (8733704142848 METADATA_ITEM 0) itemoff 11696 itemsize 33
		refs 1 gen 1496691 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 139 key (8733704159232 METADATA_ITEM 0) itemoff 11663 itemsize 33
		refs 1 gen 1496691 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 140 key (8733704241152 METADATA_ITEM 0) itemoff 11630 itemsize 33
		refs 1 gen 441944 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 141 key (8733704257536 METADATA_ITEM 0) itemoff 11597 itemsize 33
		refs 1 gen 1514029 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 142 key (8733704273920 METADATA_ITEM 0) itemoff 11564 itemsize 33
		refs 1 gen 904814 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 143 key (8733704290304 METADATA_ITEM 0) itemoff 11531 itemsize 33
		refs 1 gen 904814 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 144 key (8733704306688 METADATA_ITEM 0) itemoff 11498 itemsize 33
		refs 1 gen 904814 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 145 key (8733704323072 METADATA_ITEM 0) itemoff 11465 itemsize 33
		refs 1 gen 904814 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 146 key (8733704355840 METADATA_ITEM 0) itemoff 11432 itemsize 33
		refs 1 gen 1526001 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 147 key (8733704404992 METADATA_ITEM 0) itemoff 11399 itemsize 33
		refs 1 gen 300521 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 148 key (8733704421376 METADATA_ITEM 0) itemoff 11366 itemsize 33
		refs 1 gen 441944 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 149 key (8733704437760 METADATA_ITEM 0) itemoff 11333 itemsize 33
		refs 1 gen 441944 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 150 key (8733704454144 METADATA_ITEM 0) itemoff 11300 itemsize 33
		refs 1 gen 441944 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 151 key (8733704470528 METADATA_ITEM 0) itemoff 11267 itemsize 33
		refs 1 gen 441944 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 152 key (8733704519680 METADATA_ITEM 0) itemoff 11234 itemsize 33
		refs 1 gen 1497583 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 153 key (8733704536064 METADATA_ITEM 0) itemoff 11201 itemsize 33
		refs 1 gen 441944 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 154 key (8733704617984 METADATA_ITEM 0) itemoff 11168 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 155 key (8733704667136 METADATA_ITEM 0) itemoff 11135 itemsize 33
		refs 1 gen 441944 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 156 key (8733704683520 METADATA_ITEM 0) itemoff 11102 itemsize 33
		refs 1 gen 300521 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 157 key (8733704716288 METADATA_ITEM 0) itemoff 11069 itemsize 33
		refs 1 gen 4166 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 158 key (8733704765440 METADATA_ITEM 0) itemoff 11036 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 159 key (8733704798208 METADATA_ITEM 0) itemoff 11003 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 160 key (8733704847360 METADATA_ITEM 0) itemoff 10970 itemsize 33
		refs 1 gen 115450 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 161 key (8733704863744 METADATA_ITEM 0) itemoff 10937 itemsize 33
		refs 1 gen 115450 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 162 key (8733704880128 METADATA_ITEM 0) itemoff 10904 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 163 key (8733704912896 METADATA_ITEM 0) itemoff 10871 itemsize 33
		refs 1 gen 115450 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 164 key (8733704945664 METADATA_ITEM 0) itemoff 10838 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 165 key (8733704962048 METADATA_ITEM 0) itemoff 10805 itemsize 33
		refs 1 gen 115450 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 166 key (8733704978432 METADATA_ITEM 0) itemoff 10772 itemsize 33
		refs 1 gen 736618 flags TREE_BLOCK
		tree block skinny level 0
		shared block backref parent 10679198629888
	item 167 key (8733705043968 METADATA_ITEM 0) itemoff 10739 itemsize 33
		refs 1 gen 1554576 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 168 key (8733705093120 METADATA_ITEM 0) itemoff 10706 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 169 key (8733705109504 METADATA_ITEM 0) itemoff 10673 itemsize 33
		refs 1 gen 1554576 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 170 key (8733705125888 METADATA_ITEM 0) itemoff 10640 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 171 key (8733705142272 METADATA_ITEM 0) itemoff 10607 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 172 key (8733705175040 METADATA_ITEM 0) itemoff 10574 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 173 key (8733705191424 METADATA_ITEM 0) itemoff 10541 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 174 key (8733705207808 METADATA_ITEM 0) itemoff 10508 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 175 key (8733705224192 METADATA_ITEM 0) itemoff 10475 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 176 key (8733705240576 METADATA_ITEM 0) itemoff 10442 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 177 key (8733705256960 METADATA_ITEM 0) itemoff 10409 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 178 key (8733705273344 METADATA_ITEM 0) itemoff 10376 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 179 key (8733705289728 METADATA_ITEM 0) itemoff 10343 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 180 key (8733705306112 METADATA_ITEM 0) itemoff 10310 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 181 key (8733705322496 METADATA_ITEM 0) itemoff 10277 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 182 key (8733705338880 METADATA_ITEM 0) itemoff 10244 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 183 key (8733705453568 METADATA_ITEM 0) itemoff 10211 itemsize 33
		refs 1 gen 68949 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 184 key (8733705469952 METADATA_ITEM 0) itemoff 10178 itemsize 33
		refs 1 gen 904828 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 185 key (8733705502720 METADATA_ITEM 0) itemoff 10145 itemsize 33
		refs 1 gen 1530500 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 186 key (8733705535488 METADATA_ITEM 0) itemoff 10112 itemsize 33
		refs 1 gen 1602171 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 187 key (8733705551872 METADATA_ITEM 0) itemoff 10079 itemsize 33
		refs 1 gen 904814 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 188 key (8733705568256 METADATA_ITEM 0) itemoff 10046 itemsize 33
		refs 1 gen 904814 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 189 key (8733705584640 METADATA_ITEM 0) itemoff 10013 itemsize 33
		refs 1 gen 1530500 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 190 key (8733705601024 METADATA_ITEM 0) itemoff 9980 itemsize 33
		refs 1 gen 1522822 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 191 key (8733705633792 METADATA_ITEM 0) itemoff 9947 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 192 key (8733705650176 METADATA_ITEM 0) itemoff 9914 itemsize 33
		refs 1 gen 1522822 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 193 key (8733705666560 METADATA_ITEM 0) itemoff 9881 itemsize 33
		refs 1 gen 1522822 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 194 key (8733705682944 METADATA_ITEM 0) itemoff 9848 itemsize 33
		refs 1 gen 904828 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 195 key (8733705699328 METADATA_ITEM 0) itemoff 9815 itemsize 33
		refs 1 gen 1514029 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 196 key (8733705715712 METADATA_ITEM 0) itemoff 9782 itemsize 33
		refs 1 gen 1522822 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
leaf 364978405376 items 56 free space 8919 generation 533 owner ROOT_TREE
leaf 364978405376 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
	item 0 key (278 INODE_ITEM 0) itemoff 16123 itemsize 160
		generation 38 transid 38 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740280.173040226 (2018-02-15 16:18:00)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 1 key (278 EXTENT_DATA 0) itemoff 16070 itemsize 53
		generation 38 type 1 (regular)
		extent data disk byte 26621038592 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 2 key (279 INODE_ITEM 0) itemoff 15910 itemsize 160
		generation 38 transid 38 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740280.173040226 (2018-02-15 16:18:00)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 3 key (279 EXTENT_DATA 0) itemoff 15857 itemsize 53
		generation 38 type 1 (regular)
		extent data disk byte 26621300736 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 4 key (280 INODE_ITEM 0) itemoff 15697 itemsize 160
		generation 38 transid 38 size 262144 nbytes 262144
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740280.173040226 (2018-02-15 16:18:00)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 5 key (280 EXTENT_DATA 0) itemoff 15644 itemsize 53
		generation 38 type 1 (regular)
		extent data disk byte 26621562880 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 6 key (281 INODE_ITEM 0) itemoff 15484 itemsize 160
		generation 40 transid 40 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740347.756116870 (2018-02-15 16:19:07)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 7 key (281 EXTENT_DATA 0) itemoff 15431 itemsize 53
		generation 40 type 1 (regular)
		extent data disk byte 30292549632 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 8 key (282 INODE_ITEM 0) itemoff 15271 itemsize 160
		generation 41 transid 41 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740380.519669272 (2018-02-15 16:19:40)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 9 key (282 EXTENT_DATA 0) itemoff 15218 itemsize 53
		generation 41 type 1 (regular)
		extent data disk byte 32163766272 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 10 key (283 INODE_ITEM 0) itemoff 15058 itemsize 160
		generation 41 transid 41 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740380.519669272 (2018-02-15 16:19:40)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 11 key (283 EXTENT_DATA 0) itemoff 15005 itemsize 53
		generation 41 type 1 (regular)
		extent data disk byte 32164028416 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 12 key (284 INODE_ITEM 0) itemoff 14845 itemsize 160
		generation 41 transid 41 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740380.519669272 (2018-02-15 16:19:40)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 13 key (284 EXTENT_DATA 0) itemoff 14792 itemsize 53
		generation 41 type 1 (regular)
		extent data disk byte 32164290560 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 14 key (285 INODE_ITEM 0) itemoff 14632 itemsize 160
		generation 42 transid 42 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740419.539136240 (2018-02-15 16:20:19)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 15 key (285 EXTENT_DATA 0) itemoff 14579 itemsize 53
		generation 42 type 1 (regular)
		extent data disk byte 34247188480 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 16 key (286 INODE_ITEM 0) itemoff 14419 itemsize 160
		generation 43 transid 43 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740452.214689894 (2018-02-15 16:20:52)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 17 key (286 EXTENT_DATA 0) itemoff 14366 itemsize 53
		generation 43 type 1 (regular)
		extent data disk byte 35423563776 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 18 key (287 INODE_ITEM 0) itemoff 14206 itemsize 160
		generation 42 transid 42 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740419.543136185 (2018-02-15 16:20:19)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 19 key (287 EXTENT_DATA 0) itemoff 14153 itemsize 53
		generation 42 type 1 (regular)
		extent data disk byte 34247712768 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 20 key (288 INODE_ITEM 0) itemoff 13993 itemsize 160
		generation 44 transid 44 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740485.94240785 (2018-02-15 16:21:25)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 21 key (288 EXTENT_DATA 0) itemoff 13940 itemsize 53
		generation 44 type 1 (regular)
		extent data disk byte 37536821248 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 22 key (289 INODE_ITEM 0) itemoff 13780 itemsize 160
		generation 45 transid 45 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740521.829739034 (2018-02-15 16:22:01)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 23 key (289 EXTENT_DATA 0) itemoff 13727 itemsize 53
		generation 45 type 1 (regular)
		extent data disk byte 39703162880 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 24 key (290 INODE_ITEM 0) itemoff 13567 itemsize 160
		generation 44 transid 44 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740485.94240785 (2018-02-15 16:21:25)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 25 key (290 EXTENT_DATA 0) itemoff 13514 itemsize 53
		generation 44 type 1 (regular)
		extent data disk byte 37537345536 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 26 key (291 INODE_ITEM 0) itemoff 13354 itemsize 160
		generation 46 transid 46 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740560.741207592 (2018-02-15 16:22:40)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 27 key (291 EXTENT_DATA 0) itemoff 13301 itemsize 53
		generation 46 type 1 (regular)
		extent data disk byte 41816502272 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 28 key (292 INODE_ITEM 0) itemoff 13141 itemsize 160
		generation 47 transid 47 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740603.752620191 (2018-02-15 16:23:23)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 29 key (292 EXTENT_DATA 0) itemoff 13088 itemsize 53
		generation 47 type 1 (regular)
		extent data disk byte 44754206720 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 30 key (293 INODE_ITEM 0) itemoff 12928 itemsize 160
		generation 47 transid 47 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740603.752620191 (2018-02-15 16:23:23)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 31 key (293 EXTENT_DATA 0) itemoff 12875 itemsize 53
		generation 47 type 1 (regular)
		extent data disk byte 44754468864 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 32 key (294 INODE_ITEM 0) itemoff 12715 itemsize 160
		generation 46 transid 46 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740560.741207592 (2018-02-15 16:22:40)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 33 key (294 EXTENT_DATA 0) itemoff 12662 itemsize 53
		generation 46 type 1 (regular)
		extent data disk byte 41817288704 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 34 key (295 INODE_ITEM 0) itemoff 12502 itemsize 160
		generation 48 transid 48 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740636.584171840 (2018-02-15 16:23:56)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 35 key (295 EXTENT_DATA 0) itemoff 12449 itemsize 53
		generation 48 type 1 (regular)
		extent data disk byte 45620813824 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 36 key (296 INODE_ITEM 0) itemoff 12289 itemsize 160
		generation 47 transid 47 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740603.756620136 (2018-02-15 16:23:23)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 37 key (296 EXTENT_DATA 0) itemoff 12236 itemsize 53
		generation 47 type 1 (regular)
		extent data disk byte 44754993152 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 38 key (297 INODE_ITEM 0) itemoff 12076 itemsize 160
		generation 47 transid 47 size 262144 nbytes 262144
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740603.756620136 (2018-02-15 16:23:23)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 39 key (297 EXTENT_DATA 0) itemoff 12023 itemsize 53
		generation 47 type 1 (regular)
		extent data disk byte 44755255296 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 40 key (298 INODE_ITEM 0) itemoff 11863 itemsize 160
		generation 49 transid 49 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740669.327724712 (2018-02-15 16:24:29)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 41 key (298 EXTENT_DATA 0) itemoff 11810 itemsize 53
		generation 49 type 1 (regular)
		extent data disk byte 46119051264 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 42 key (299 INODE_ITEM 0) itemoff 11650 itemsize 160
		generation 51 transid 51 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740736.874802394 (2018-02-15 16:25:36)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 43 key (299 EXTENT_DATA 0) itemoff 11597 itemsize 53
		generation 51 type 1 (regular)
		extent data disk byte 47770337280 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 44 key (300 INODE_ITEM 0) itemoff 11437 itemsize 160
		generation 53 transid 53 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740812.653767782 (2018-02-15 16:26:52)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 45 key (300 EXTENT_DATA 0) itemoff 11384 itemsize 53
		generation 53 type 1 (regular)
		extent data disk byte 52100096000 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 46 key (301 INODE_ITEM 0) itemoff 11224 itemsize 160
		generation 52 transid 52 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740777.838243105 (2018-02-15 16:26:17)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 47 key (301 EXTENT_DATA 0) itemoff 11171 itemsize 53
		generation 52 type 1 (regular)
		extent data disk byte 50697764864 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 48 key (302 INODE_ITEM 0) itemoff 11011 itemsize 160
		generation 52 transid 52 size 262144 nbytes 262144
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740777.838243105 (2018-02-15 16:26:17)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 49 key (302 EXTENT_DATA 0) itemoff 10958 itemsize 53
		generation 52 type 1 (regular)
		extent data disk byte 50698027008 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 50 key (303 INODE_ITEM 0) itemoff 10798 itemsize 160
		generation 54 transid 54 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740843.369348452 (2018-02-15 16:27:23)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 51 key (303 EXTENT_DATA 0) itemoff 10745 itemsize 53
		generation 54 type 1 (regular)
		extent data disk byte 53655924736 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 52 key (304 INODE_ITEM 0) itemoff 10585 itemsize 160
		generation 55 transid 55 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740874.100928922 (2018-02-15 16:27:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 53 key (304 EXTENT_DATA 0) itemoff 10532 itemsize 53
		generation 55 type 1 (regular)
		extent data disk byte 55110717440 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 54 key (305 INODE_ITEM 0) itemoff 10372 itemsize 160
		generation 54 transid 54 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740843.369348452 (2018-02-15 16:27:23)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 55 key (305 EXTENT_DATA 0) itemoff 10319 itemsize 53
		generation 54 type 1 (regular)
		extent data disk byte 53656453120 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
parent transid verify failed on 13577300590592 wanted 1601950 found 1602229
leaf 13577300590592 items 119 free space 581 generation 1601950 owner ROOT_TREE
leaf 13577300590592 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
	item 0 key (306 INODE_ITEM 0) itemoff 16123 itemsize 160
		generation 56 transid 56 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740904.920508209 (2018-02-15 16:28:24)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 1 key (306 EXTENT_DATA 0) itemoff 16070 itemsize 53
		generation 56 type 1 (regular)
		extent data disk byte 55728295936 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 2 key (307 INODE_ITEM 0) itemoff 15910 itemsize 160
		generation 57 transid 57 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740937.620061853 (2018-02-15 16:28:57)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 3 key (307 EXTENT_DATA 0) itemoff 15857 itemsize 53
		generation 57 type 1 (regular)
		extent data disk byte 56279109632 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 4 key (308 INODE_ITEM 0) itemoff 15697 itemsize 160
		generation 59 transid 59 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518740999.155221937 (2018-02-15 16:29:59)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 5 key (308 EXTENT_DATA 0) itemoff 15644 itemsize 53
		generation 59 type 1 (regular)
		extent data disk byte 58089660416 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 6 key (309 INODE_ITEM 0) itemoff 15484 itemsize 160
		generation 60 transid 60 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741031.798776402 (2018-02-15 16:30:31)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 7 key (309 EXTENT_DATA 0) itemoff 15431 itemsize 53
		generation 60 type 1 (regular)
		extent data disk byte 59047624704 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 8 key (310 INODE_ITEM 0) itemoff 15271 itemsize 160
		generation 61 transid 61 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741064.558329304 (2018-02-15 16:31:04)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 9 key (310 EXTENT_DATA 0) itemoff 15218 itemsize 53
		generation 61 type 1 (regular)
		extent data disk byte 60360396800 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 10 key (311 INODE_ITEM 0) itemoff 15058 itemsize 160
		generation 1601940 transid 1601940 size 131072 nbytes 59050164224
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 450517 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1648366373.835066464 (2022-03-27 00:32:53)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 11 key (311 EXTENT_DATA 0) itemoff 15005 itemsize 53
		generation 1601940 type 1 (regular)
		extent data disk byte 4552318590976 nr 131072
		extent data offset 0 nr 131072 ram 131072
		extent compression 0 (none)
	item 12 key (312 INODE_ITEM 0) itemoff 14845 itemsize 160
		generation 62 transid 62 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741097.333882006 (2018-02-15 16:31:37)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 13 key (312 EXTENT_DATA 0) itemoff 14792 itemsize 53
		generation 62 type 1 (regular)
		extent data disk byte 61363699712 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 14 key (313 INODE_ITEM 0) itemoff 14632 itemsize 160
		generation 63 transid 63 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741132.157406780 (2018-02-15 16:32:12)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 15 key (313 EXTENT_DATA 0) itemoff 14579 itemsize 53
		generation 63 type 1 (regular)
		extent data disk byte 62069760000 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 16 key (314 INODE_ITEM 0) itemoff 14419 itemsize 160
		generation 64 transid 64 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741164.920959686 (2018-02-15 16:32:44)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 17 key (314 EXTENT_DATA 0) itemoff 14366 itemsize 53
		generation 64 type 1 (regular)
		extent data disk byte 62734540800 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 18 key (315 INODE_ITEM 0) itemoff 14206 itemsize 160
		generation 69 transid 69 size 262144 nbytes 1572864
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 6 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741339.2584470 (2018-02-15 16:35:39)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 19 key (315 EXTENT_DATA 0) itemoff 14153 itemsize 53
		generation 69 type 1 (regular)
		extent data disk byte 69593169920 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 20 key (316 INODE_ITEM 0) itemoff 13993 itemsize 160
		generation 65 transid 65 size 262144 nbytes 262144
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741197.684512612 (2018-02-15 16:33:17)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 21 key (316 EXTENT_DATA 0) itemoff 13940 itemsize 53
		generation 65 type 1 (regular)
		extent data disk byte 63950589952 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 22 key (317 INODE_ITEM 0) itemoff 13780 itemsize 160
		generation 69 transid 69 size 262144 nbytes 1310720
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741339.2584470 (2018-02-15 16:35:39)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 23 key (317 EXTENT_DATA 0) itemoff 13727 itemsize 53
		generation 69 type 1 (regular)
		extent data disk byte 69593956352 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 24 key (318 INODE_ITEM 0) itemoff 13567 itemsize 160
		generation 67 transid 67 size 262144 nbytes 262144
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741265.271590412 (2018-02-15 16:34:25)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 25 key (318 EXTENT_DATA 0) itemoff 13514 itemsize 53
		generation 67 type 1 (regular)
		extent data disk byte 66273447936 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 26 key (319 INODE_ITEM 0) itemoff 13354 itemsize 160
		generation 69 transid 69 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741339.2584470 (2018-02-15 16:35:39)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 27 key (319 EXTENT_DATA 0) itemoff 13301 itemsize 53
		generation 69 type 1 (regular)
		extent data disk byte 69594218496 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 28 key (320 INODE_ITEM 0) itemoff 13141 itemsize 160
		generation 87 transid 87 size 262144 nbytes 2097152
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 8 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741855.83545862 (2018-02-15 16:44:15)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 29 key (320 EXTENT_DATA 0) itemoff 13088 itemsize 53
		generation 87 type 1 (regular)
		extent data disk byte 79485554688 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 30 key (321 INODE_ITEM 0) itemoff 12928 itemsize 160
		generation 69 transid 69 size 262144 nbytes 262144
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741339.2584470 (2018-02-15 16:35:39)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 31 key (321 EXTENT_DATA 0) itemoff 12875 itemsize 53
		generation 69 type 1 (regular)
		extent data disk byte 69593694208 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 32 key (322 INODE_ITEM 0) itemoff 12715 itemsize 160
		generation 86 transid 86 size 262144 nbytes 2097152
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 8 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741823.343978624 (2018-02-15 16:43:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 33 key (322 EXTENT_DATA 0) itemoff 12662 itemsize 53
		generation 86 type 1 (regular)
		extent data disk byte 76264198144 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 34 key (323 INODE_ITEM 0) itemoff 12502 itemsize 160
		generation 86 transid 86 size 262144 nbytes 2359296
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741823.347978570 (2018-02-15 16:43:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 35 key (323 EXTENT_DATA 0) itemoff 12449 itemsize 53
		generation 86 type 1 (regular)
		extent data disk byte 76264591360 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 36 key (324 INODE_ITEM 0) itemoff 12289 itemsize 160
		generation 76 transid 76 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741568.379455572 (2018-02-15 16:39:28)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 37 key (324 EXTENT_DATA 0) itemoff 12236 itemsize 53
		generation 76 type 1 (regular)
		extent data disk byte 73456918528 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 38 key (325 INODE_ITEM 0) itemoff 12076 itemsize 160
		generation 86 transid 86 size 262144 nbytes 2359296
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741823.347978570 (2018-02-15 16:43:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 39 key (325 EXTENT_DATA 0) itemoff 12023 itemsize 53
		generation 86 type 1 (regular)
		extent data disk byte 77337407488 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 40 key (326 INODE_ITEM 0) itemoff 11863 itemsize 160
		generation 1601950 transid 1601950 size 131072 nbytes 54084239360
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 412630 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1648367018.70270714 (2022-03-27 00:43:38)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 41 key (326 EXTENT_DATA 0) itemoff 11810 itemsize 53
		generation 1601950 type 1 (regular)
		extent data disk byte 937112211456 nr 131072
		extent data offset 0 nr 131072 ram 131072
		extent compression 0 (none)
	item 42 key (327 INODE_ITEM 0) itemoff 11650 itemsize 160
		generation 86 transid 86 size 262144 nbytes 2359296
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741823.347978570 (2018-02-15 16:43:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 43 key (327 EXTENT_DATA 0) itemoff 11597 itemsize 53
		generation 86 type 1 (regular)
		extent data disk byte 77337669632 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 44 key (328 INODE_ITEM 0) itemoff 11437 itemsize 160
		generation 1448946 transid 1448946 size 262144 nbytes 28573696
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 109 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699040.284833350 (2020-07-13 20:57:20)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 45 key (328 EXTENT_DATA 0) itemoff 11384 itemsize 53
		generation 1448946 type 1 (regular)
		extent data disk byte 385896763392 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 46 key (329 INODE_ITEM 0) itemoff 11224 itemsize 160
		generation 88 transid 88 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741889.903071124 (2018-02-15 16:44:49)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 47 key (329 EXTENT_DATA 0) itemoff 11171 itemsize 53
		generation 88 type 1 (regular)
		extent data disk byte 81740009472 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 48 key (331 INODE_ITEM 0) itemoff 11011 itemsize 160
		generation 87 transid 87 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741855.83545862 (2018-02-15 16:44:15)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 49 key (331 EXTENT_DATA 0) itemoff 10958 itemsize 53
		generation 87 type 1 (regular)
		extent data disk byte 80142901248 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 50 key (332 INODE_ITEM 0) itemoff 10798 itemsize 160
		generation 89 transid 89 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741922.666624432 (2018-02-15 16:45:22)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 51 key (332 EXTENT_DATA 0) itemoff 10745 itemsize 53
		generation 89 type 1 (regular)
		extent data disk byte 83344203776 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 52 key (333 INODE_ITEM 0) itemoff 10585 itemsize 160
		generation 90 transid 90 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741957.478149836 (2018-02-15 16:45:57)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 53 key (333 EXTENT_DATA 0) itemoff 10532 itemsize 53
		generation 90 type 1 (regular)
		extent data disk byte 84745506816 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 54 key (334 INODE_ITEM 0) itemoff 10372 itemsize 160
		generation 1420 transid 1420 size 262144 nbytes 17301504
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 66 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518797932.463632141 (2018-02-16 08:18:52)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 55 key (334 EXTENT_DATA 0) itemoff 10319 itemsize 53
		generation 1420 type 1 (regular)
		extent data disk byte 156876902400 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 56 key (335 INODE_ITEM 0) itemoff 10159 itemsize 160
		generation 90 transid 90 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741957.482149782 (2018-02-15 16:45:57)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 57 key (335 EXTENT_DATA 0) itemoff 10106 itemsize 53
		generation 90 type 1 (regular)
		extent data disk byte 84746031104 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 58 key (336 INODE_ITEM 0) itemoff 9946 itemsize 160
		generation 91 transid 91 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518741992.301675095 (2018-02-15 16:46:32)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 59 key (336 EXTENT_DATA 0) itemoff 9893 itemsize 53
		generation 91 type 1 (regular)
		extent data disk byte 85901545472 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 60 key (337 INODE_ITEM 0) itemoff 9733 itemsize 160
		generation 101 transid 101 size 262144 nbytes 3145728
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742342.504901802 (2018-02-15 16:52:22)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 61 key (337 EXTENT_DATA 0) itemoff 9680 itemsize 53
		generation 101 type 1 (regular)
		extent data disk byte 85902106624 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 62 key (338 INODE_ITEM 0) itemoff 9520 itemsize 160
		generation 1475 transid 1475 size 262144 nbytes 13369344
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 51 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518800228.376440697 (2018-02-16 08:57:08)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 63 key (338 EXTENT_DATA 0) itemoff 9467 itemsize 53
		generation 1475 type 1 (regular)
		extent data disk byte 1706741551104 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 64 key (339 INODE_ITEM 0) itemoff 9307 itemsize 160
		generation 104 transid 104 size 262144 nbytes 3145728
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742440.807562229 (2018-02-15 16:54:00)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 65 key (339 EXTENT_DATA 0) itemoff 9254 itemsize 53
		generation 104 type 1 (regular)
		extent data disk byte 95592022016 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 66 key (340 INODE_ITEM 0) itemoff 9094 itemsize 160
		generation 104 transid 104 size 262144 nbytes 1835008
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 7 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742440.811562175 (2018-02-15 16:54:00)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 67 key (340 EXTENT_DATA 0) itemoff 9041 itemsize 53
		generation 104 type 1 (regular)
		extent data disk byte 96665616384 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 68 key (341 INODE_ITEM 0) itemoff 8881 itemsize 160
		generation 104 transid 104 size 262144 nbytes 1835008
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 7 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742440.811562175 (2018-02-15 16:54:00)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 69 key (341 EXTENT_DATA 0) itemoff 8828 itemsize 53
		generation 104 type 1 (regular)
		extent data disk byte 97739059200 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 70 key (342 INODE_ITEM 0) itemoff 8668 itemsize 160
		generation 104 transid 104 size 262144 nbytes 1835008
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 7 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742440.811562175 (2018-02-15 16:54:00)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 71 key (342 EXTENT_DATA 0) itemoff 8615 itemsize 53
		generation 104 type 1 (regular)
		extent data disk byte 97739583488 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 72 key (343 INODE_ITEM 0) itemoff 8455 itemsize 160
		generation 105 transid 105 size 262144 nbytes 2097152
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 8 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742481.771004057 (2018-02-15 16:54:41)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 73 key (343 EXTENT_DATA 0) itemoff 8402 itemsize 53
		generation 105 type 1 (regular)
		extent data disk byte 109406760960 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 74 key (344 INODE_ITEM 0) itemoff 8242 itemsize 160
		generation 104 transid 104 size 262144 nbytes 1310720
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742440.811562175 (2018-02-15 16:54:00)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 75 key (344 EXTENT_DATA 0) itemoff 8189 itemsize 53
		generation 104 type 1 (regular)
		extent data disk byte 99886931968 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 76 key (345 INODE_ITEM 0) itemoff 8029 itemsize 160
		generation 104 transid 104 size 262144 nbytes 1835008
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 7 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742440.811562175 (2018-02-15 16:54:00)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 77 key (345 EXTENT_DATA 0) itemoff 7976 itemsize 53
		generation 104 type 1 (regular)
		extent data disk byte 100959903744 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 78 key (346 INODE_ITEM 0) itemoff 7816 itemsize 160
		generation 106 transid 106 size 262144 nbytes 2359296
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742514.534557633 (2018-02-15 16:55:14)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 79 key (346 EXTENT_DATA 0) itemoff 7763 itemsize 53
		generation 106 type 1 (regular)
		extent data disk byte 113845682176 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 80 key (347 INODE_ITEM 0) itemoff 7603 itemsize 160
		generation 106 transid 106 size 262144 nbytes 2359296
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742514.538557578 (2018-02-15 16:55:14)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 81 key (347 EXTENT_DATA 0) itemoff 7550 itemsize 53
		generation 106 type 1 (regular)
		extent data disk byte 115284467712 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 82 key (348 INODE_ITEM 0) itemoff 7390 itemsize 160
		generation 106 transid 106 size 262144 nbytes 2359296
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742514.538557578 (2018-02-15 16:55:14)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 83 key (348 EXTENT_DATA 0) itemoff 7337 itemsize 53
		generation 106 type 1 (regular)
		extent data disk byte 115284729856 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 84 key (349 INODE_ITEM 0) itemoff 7177 itemsize 160
		generation 107 transid 107 size 262144 nbytes 2359296
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742549.354083210 (2018-02-15 16:55:49)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 85 key (349 EXTENT_DATA 0) itemoff 7124 itemsize 53
		generation 107 type 1 (regular)
		extent data disk byte 113621704704 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 86 key (350 INODE_ITEM 0) itemoff 6964 itemsize 160
		generation 107 transid 107 size 262144 nbytes 1310720
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742549.354083210 (2018-02-15 16:55:49)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 87 key (350 EXTENT_DATA 0) itemoff 6911 itemsize 53
		generation 107 type 1 (regular)
		extent data disk byte 113623695360 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 88 key (351 INODE_ITEM 0) itemoff 6751 itemsize 160
		generation 106 transid 106 size 262144 nbytes 2359296
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742514.538557578 (2018-02-15 16:55:14)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 89 key (351 EXTENT_DATA 0) itemoff 6698 itemsize 53
		generation 106 type 1 (regular)
		extent data disk byte 115285516288 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 90 key (352 INODE_ITEM 0) itemoff 6538 itemsize 160
		generation 106 transid 106 size 262144 nbytes 2359296
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742514.538557578 (2018-02-15 16:55:14)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 91 key (352 EXTENT_DATA 0) itemoff 6485 itemsize 53
		generation 106 type 1 (regular)
		extent data disk byte 115285778432 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 92 key (353 INODE_ITEM 0) itemoff 6325 itemsize 160
		generation 107 transid 107 size 262144 nbytes 1572864
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 6 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742549.354083210 (2018-02-15 16:55:49)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 93 key (353 EXTENT_DATA 0) itemoff 6272 itemsize 53
		generation 107 type 1 (regular)
		extent data disk byte 115993206784 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 94 key (354 INODE_ITEM 0) itemoff 6112 itemsize 160
		generation 108 transid 108 size 262144 nbytes 1310720
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742582.121636759 (2018-02-15 16:56:22)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 95 key (354 EXTENT_DATA 0) itemoff 6059 itemsize 53
		generation 108 type 1 (regular)
		extent data disk byte 118523154432 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 96 key (355 INODE_ITEM 0) itemoff 5899 itemsize 160
		generation 103 transid 103 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742408.48008631 (2018-02-15 16:53:28)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 97 key (355 EXTENT_DATA 0) itemoff 5846 itemsize 53
		generation 103 type 1 (regular)
		extent data disk byte 109407629312 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 98 key (356 INODE_ITEM 0) itemoff 5686 itemsize 160
		generation 107 transid 107 size 262144 nbytes 2097152
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 8 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742549.354083210 (2018-02-15 16:55:49)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 99 key (356 EXTENT_DATA 0) itemoff 5633 itemsize 53
		generation 107 type 1 (regular)
		extent data disk byte 116802277376 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 100 key (357 INODE_ITEM 0) itemoff 5473 itemsize 160
		generation 102 transid 102 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742375.276455209 (2018-02-15 16:52:55)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 101 key (357 EXTENT_DATA 0) itemoff 5420 itemsize 53
		generation 102 type 1 (regular)
		extent data disk byte 90223198208 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 102 key (358 INODE_ITEM 0) itemoff 5260 itemsize 160
		generation 107 transid 107 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742549.354083210 (2018-02-15 16:55:49)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 103 key (358 EXTENT_DATA 0) itemoff 5207 itemsize 53
		generation 107 type 1 (regular)
		extent data disk byte 116802539520 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 104 key (359 INODE_ITEM 0) itemoff 5047 itemsize 160
		generation 107 transid 107 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742549.354083210 (2018-02-15 16:55:49)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 105 key (359 EXTENT_DATA 0) itemoff 4994 itemsize 53
		generation 107 type 1 (regular)
		extent data disk byte 116802801664 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 106 key (360 INODE_ITEM 0) itemoff 4834 itemsize 160
		generation 1453981 transid 1453981 size 262144 nbytes 8388608
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 32 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594951634.324611316 (2020-07-16 19:07:14)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 107 key (360 EXTENT_DATA 0) itemoff 4781 itemsize 53
		generation 1453981 type 1 (regular)
		extent data disk byte 749642706944 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 108 key (361 INODE_ITEM 0) itemoff 4621 itemsize 160
		generation 109 transid 109 size 262144 nbytes 1572864
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 6 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742616.937162420 (2018-02-15 16:56:56)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 109 key (361 EXTENT_DATA 0) itemoff 4568 itemsize 53
		generation 109 type 1 (regular)
		extent data disk byte 119953424384 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 110 key (362 INODE_ITEM 0) itemoff 4408 itemsize 160
		generation 108 transid 108 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742582.125636704 (2018-02-15 16:56:22)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 111 key (362 EXTENT_DATA 0) itemoff 4355 itemsize 53
		generation 108 type 1 (regular)
		extent data disk byte 118523940864 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 112 key (363 INODE_ITEM 0) itemoff 4195 itemsize 160
		generation 108 transid 108 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742582.125636704 (2018-02-15 16:56:22)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 113 key (363 EXTENT_DATA 0) itemoff 4142 itemsize 53
		generation 108 type 1 (regular)
		extent data disk byte 118524203008 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 114 key (364 INODE_ITEM 0) itemoff 3982 itemsize 160
		generation 109 transid 109 size 262144 nbytes 1310720
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742616.937162420 (2018-02-15 16:56:56)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 115 key (364 EXTENT_DATA 0) itemoff 3929 itemsize 53
		generation 109 type 1 (regular)
		extent data disk byte 119953686528 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 116 key (365 INODE_ITEM 0) itemoff 3769 itemsize 160
		generation 108 transid 108 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742582.125636704 (2018-02-15 16:56:22)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 117 key (365 EXTENT_DATA 0) itemoff 3716 itemsize 53
		generation 108 type 1 (regular)
		extent data disk byte 118524727296 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 118 key (366 INODE_ITEM 0) itemoff 3556 itemsize 160
		generation 110 transid 110 size 262144 nbytes 1310720
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742647.652743952 (2018-02-15 16:57:27)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
leaf 11822267351040 items 123 free space 162 generation 1591706 owner ROOT_TREE
leaf 11822267351040 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
	item 0 key (366 EXTENT_DATA 0) itemoff 16230 itemsize 53
		generation 110 type 1 (regular)
		extent data disk byte 121487896576 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 1 key (367 INODE_ITEM 0) itemoff 16070 itemsize 160
		generation 109 transid 109 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742616.937162420 (2018-02-15 16:56:56)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 2 key (367 EXTENT_DATA 0) itemoff 16017 itemsize 53
		generation 109 type 1 (regular)
		extent data disk byte 119954210816 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 3 key (368 INODE_ITEM 0) itemoff 15857 itemsize 160
		generation 109 transid 109 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742616.941162365 (2018-02-15 16:56:56)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 4 key (368 EXTENT_DATA 0) itemoff 15804 itemsize 53
		generation 109 type 1 (regular)
		extent data disk byte 119954472960 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 5 key (369 INODE_ITEM 0) itemoff 15644 itemsize 160
		generation 1475 transid 1475 size 262144 nbytes 14155776
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 54 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518800228.384440588 (2018-02-16 08:57:08)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 6 key (369 EXTENT_DATA 0) itemoff 15591 itemsize 53
		generation 1475 type 1 (regular)
		extent data disk byte 1728215785472 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 7 key (370 INODE_ITEM 0) itemoff 15431 itemsize 160
		generation 111 transid 111 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742678.376325386 (2018-02-15 16:57:58)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 8 key (370 EXTENT_DATA 0) itemoff 15378 itemsize 53
		generation 111 type 1 (regular)
		extent data disk byte 123032813568 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 9 key (371 INODE_ITEM 0) itemoff 15218 itemsize 160
		generation 112 transid 112 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742711.143878988 (2018-02-15 16:58:31)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 10 key (371 EXTENT_DATA 0) itemoff 15165 itemsize 53
		generation 112 type 1 (regular)
		extent data disk byte 124644073472 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 11 key (372 INODE_ITEM 0) itemoff 15005 itemsize 160
		generation 112 transid 112 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742711.147878934 (2018-02-15 16:58:31)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 12 key (372 EXTENT_DATA 0) itemoff 14952 itemsize 53
		generation 112 type 1 (regular)
		extent data disk byte 124644335616 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 13 key (373 INODE_ITEM 0) itemoff 14792 itemsize 160
		generation 1448951 transid 1448951 size 262144 nbytes 16252928
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 62 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699242.6790619 (2020-07-13 21:00:42)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 14 key (373 EXTENT_DATA 0) itemoff 14739 itemsize 53
		generation 1448951 type 1 (regular)
		extent data disk byte 385906561024 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 15 key (374 INODE_ITEM 0) itemoff 14579 itemsize 160
		generation 112 transid 112 size 262144 nbytes 262144
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742711.147878934 (2018-02-15 16:58:31)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 16 key (374 EXTENT_DATA 0) itemoff 14526 itemsize 53
		generation 112 type 1 (regular)
		extent data disk byte 124644859904 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 17 key (375 INODE_ITEM 0) itemoff 14366 itemsize 160
		generation 114 transid 114 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742776.678986230 (2018-02-15 16:59:36)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 18 key (375 EXTENT_DATA 0) itemoff 14313 itemsize 53
		generation 114 type 1 (regular)
		extent data disk byte 127791153152 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 19 key (376 INODE_ITEM 0) itemoff 14153 itemsize 160
		generation 1448958 transid 1448958 size 262144 nbytes 4718592
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 18 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699388.446423495 (2020-07-13 21:03:08)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 20 key (376 EXTENT_DATA 0) itemoff 14100 itemsize 53
		generation 1448958 type 1 (regular)
		extent data disk byte 385917460480 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 21 key (377 INODE_ITEM 0) itemoff 13940 itemsize 160
		generation 116 transid 116 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742840.170121365 (2018-02-15 17:00:40)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 22 key (377 EXTENT_DATA 0) itemoff 13887 itemsize 53
		generation 116 type 1 (regular)
		extent data disk byte 130674802688 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 23 key (378 INODE_ITEM 0) itemoff 13727 itemsize 160
		generation 1448946 transid 1448946 size 262144 nbytes 18612224
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 71 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699040.284833350 (2020-07-13 20:57:20)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 24 key (378 EXTENT_DATA 0) itemoff 13674 itemsize 53
		generation 1448946 type 1 (regular)
		extent data disk byte 385897598976 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 25 key (379 INODE_ITEM 0) itemoff 13514 itemsize 160
		generation 117 transid 117 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742870.889702925 (2018-02-15 17:01:10)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 26 key (379 EXTENT_DATA 0) itemoff 13461 itemsize 53
		generation 117 type 1 (regular)
		extent data disk byte 132128493568 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 27 key (380 INODE_ITEM 0) itemoff 13301 itemsize 160
		generation 118 transid 118 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742901.605284551 (2018-02-15 17:01:41)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 28 key (380 EXTENT_DATA 0) itemoff 13248 itemsize 53
		generation 118 type 1 (regular)
		extent data disk byte 133475274752 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 29 key (381 INODE_ITEM 0) itemoff 13088 itemsize 160
		generation 119 transid 119 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742934.376838184 (2018-02-15 17:02:14)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 30 key (381 EXTENT_DATA 0) itemoff 13035 itemsize 53
		generation 119 type 1 (regular)
		extent data disk byte 134998917120 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 31 key (382 INODE_ITEM 0) itemoff 12875 itemsize 160
		generation 1448948 transid 1448948 size 262144 nbytes 17825792
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 68 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699122.230374758 (2020-07-13 20:58:42)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 32 key (382 EXTENT_DATA 0) itemoff 12822 itemsize 53
		generation 1448948 type 1 (regular)
		extent data disk byte 385902223360 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 33 key (383 INODE_ITEM 0) itemoff 12662 itemsize 160
		generation 121 transid 121 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742995.820001329 (2018-02-15 17:03:15)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 34 key (383 EXTENT_DATA 0) itemoff 12609 itemsize 53
		generation 121 type 1 (regular)
		extent data disk byte 138572734464 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 35 key (384 INODE_ITEM 0) itemoff 12449 itemsize 160
		generation 1448962 transid 1448962 size 262144 nbytes 6291456
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 24 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699552.281556333 (2020-07-13 21:05:52)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 36 key (384 EXTENT_DATA 0) itemoff 12396 itemsize 53
		generation 1448962 type 1 (regular)
		extent data disk byte 385924395008 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 37 key (385 INODE_ITEM 0) itemoff 12236 itemsize 160
		generation 122 transid 122 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743030.635527161 (2018-02-15 17:03:50)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 38 key (385 EXTENT_DATA 0) itemoff 12183 itemsize 53
		generation 122 type 1 (regular)
		extent data disk byte 140095905792 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 39 key (386 INODE_ITEM 0) itemoff 12023 itemsize 160
		generation 121 transid 121 size 262144 nbytes 262144
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518742995.820001329 (2018-02-15 17:03:15)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 40 key (386 EXTENT_DATA 0) itemoff 11970 itemsize 53
		generation 121 type 1 (regular)
		extent data disk byte 138573520896 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 41 key (387 INODE_ITEM 0) itemoff 11810 itemsize 160
		generation 1448948 transid 1448948 size 262144 nbytes 5242880
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 20 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699122.230374758 (2020-07-13 20:58:42)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 42 key (387 EXTENT_DATA 0) itemoff 11757 itemsize 53
		generation 1448948 type 1 (regular)
		extent data disk byte 385902485504 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 43 key (388 INODE_ITEM 0) itemoff 11597 itemsize 160
		generation 124 transid 124 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743092.70690480 (2018-02-15 17:04:52)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 44 key (388 EXTENT_DATA 0) itemoff 11544 itemsize 53
		generation 124 type 1 (regular)
		extent data disk byte 143136792576 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 45 key (389 INODE_ITEM 0) itemoff 11384 itemsize 160
		generation 125 transid 125 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743122.790272128 (2018-02-15 17:05:22)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 46 key (389 EXTENT_DATA 0) itemoff 11331 itemsize 53
		generation 125 type 1 (regular)
		extent data disk byte 144674807808 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 47 key (390 INODE_ITEM 0) itemoff 11171 itemsize 160
		generation 1449887 transid 1449887 size 262144 nbytes 5505024
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 21 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594782921.334009642 (2020-07-14 20:15:21)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 48 key (390 EXTENT_DATA 0) itemoff 11118 itemsize 53
		generation 1449887 type 1 (regular)
		extent data disk byte 721578090496 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 49 key (391 INODE_ITEM 0) itemoff 10958 itemsize 160
		generation 126 transid 126 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743155.585825517 (2018-02-15 17:05:55)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 50 key (391 EXTENT_DATA 0) itemoff 10905 itemsize 53
		generation 126 type 1 (regular)
		extent data disk byte 146200510464 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 51 key (392 INODE_ITEM 0) itemoff 10745 itemsize 160
		generation 127 transid 127 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743186.281407513 (2018-02-15 17:06:26)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 52 key (392 EXTENT_DATA 0) itemoff 10692 itemsize 53
		generation 127 type 1 (regular)
		extent data disk byte 147668541440 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 53 key (393 INODE_ITEM 0) itemoff 10532 itemsize 160
		generation 1448949 transid 1448949 size 262144 nbytes 4194304
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 16 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699168.280995440 (2020-07-13 20:59:28)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 54 key (393 EXTENT_DATA 0) itemoff 10479 itemsize 53
		generation 1448949 type 1 (regular)
		extent data disk byte 385891045376 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 55 key (394 INODE_ITEM 0) itemoff 10319 itemsize 160
		generation 128 transid 128 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743216.996989248 (2018-02-15 17:06:56)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 56 key (394 EXTENT_DATA 0) itemoff 10266 itemsize 53
		generation 128 type 1 (regular)
		extent data disk byte 149261901824 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 57 key (395 INODE_ITEM 0) itemoff 10106 itemsize 160
		generation 129 transid 129 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743247.720570883 (2018-02-15 17:07:27)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 58 key (395 EXTENT_DATA 0) itemoff 10053 itemsize 53
		generation 129 type 1 (regular)
		extent data disk byte 150720843776 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 59 key (396 INODE_ITEM 0) itemoff 9893 itemsize 160
		generation 1448965 transid 1448965 size 262144 nbytes 5505024
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 21 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699675.185917664 (2020-07-13 21:07:55)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 60 key (396 EXTENT_DATA 0) itemoff 9840 itemsize 53
		generation 1448965 type 1 (regular)
		extent data disk byte 385926844416 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 61 key (397 INODE_ITEM 0) itemoff 9680 itemsize 160
		generation 130 transid 130 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743278.444152528 (2018-02-15 17:07:58)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 62 key (397 EXTENT_DATA 0) itemoff 9627 itemsize 53
		generation 130 type 1 (regular)
		extent data disk byte 151962271744 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 63 key (398 INODE_ITEM 0) itemoff 9467 itemsize 160
		generation 1448956 transid 1448956 size 262144 nbytes 4718592
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 18 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699294.481223871 (2020-07-13 21:01:34)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 64 key (398 EXTENT_DATA 0) itemoff 9414 itemsize 53
		generation 1448956 type 1 (regular)
		extent data disk byte 385908002816 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 65 key (399 INODE_ITEM 0) itemoff 9254 itemsize 160
		generation 132 transid 132 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743343.975260243 (2018-02-15 17:09:03)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 66 key (399 EXTENT_DATA 0) itemoff 9201 itemsize 53
		generation 132 type 1 (regular)
		extent data disk byte 155183456256 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 67 key (400 INODE_ITEM 0) itemoff 9041 itemsize 160
		generation 1448967 transid 1448967 size 262144 nbytes 7864320
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 30 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699747.867770777 (2020-07-13 21:09:07)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 68 key (400 EXTENT_DATA 0) itemoff 8988 itemsize 53
		generation 1448967 type 1 (regular)
		extent data disk byte 385927790592 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 69 key (401 INODE_ITEM 0) itemoff 8828 itemsize 160
		generation 134 transid 134 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743409.510367947 (2018-02-15 17:10:09)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 70 key (401 EXTENT_DATA 0) itemoff 8775 itemsize 53
		generation 134 type 1 (regular)
		extent data disk byte 158500700160 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 71 key (402 INODE_ITEM 0) itemoff 8615 itemsize 160
		generation 134 transid 134 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743409.514367892 (2018-02-15 17:10:09)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 72 key (402 EXTENT_DATA 0) itemoff 8562 itemsize 53
		generation 134 type 1 (regular)
		extent data disk byte 158500962304 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 73 key (403 INODE_ITEM 0) itemoff 8402 itemsize 160
		generation 135 transid 135 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743442.277921815 (2018-02-15 17:10:42)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 74 key (403 EXTENT_DATA 0) itemoff 8349 itemsize 53
		generation 135 type 1 (regular)
		extent data disk byte 160101437440 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 75 key (404 INODE_ITEM 0) itemoff 8189 itemsize 160
		generation 1420 transid 1420 size 262144 nbytes 3407872
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 13 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518797935.239594272 (2018-02-16 08:18:55)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 76 key (404 EXTENT_DATA 0) itemoff 8136 itemsize 53
		generation 1420 type 1 (regular)
		extent data disk byte 2149658714112 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 77 key (405 INODE_ITEM 0) itemoff 7976 itemsize 160
		generation 134 transid 134 size 262144 nbytes 262144
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743409.514367892 (2018-02-15 17:10:09)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 78 key (405 EXTENT_DATA 0) itemoff 7923 itemsize 53
		generation 134 type 1 (regular)
		extent data disk byte 158501748736 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 79 key (406 INODE_ITEM 0) itemoff 7763 itemsize 160
		generation 136 transid 136 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743475.49475641 (2018-02-15 17:11:15)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 80 key (406 EXTENT_DATA 0) itemoff 7710 itemsize 53
		generation 136 type 1 (regular)
		extent data disk byte 161595084800 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 81 key (407 INODE_ITEM 0) itemoff 7550 itemsize 160
		generation 2368 transid 2368 size 262144 nbytes 53215232
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 203 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518841610.511576836 (2018-02-16 20:26:50)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 82 key (407 EXTENT_DATA 0) itemoff 7497 itemsize 53
		generation 2368 type 1 (regular)
		extent data disk byte 1580039909376 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 83 key (408 INODE_ITEM 0) itemoff 7337 itemsize 160
		generation 138 transid 138 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743540.584583433 (2018-02-15 17:12:20)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 84 key (408 EXTENT_DATA 0) itemoff 7284 itemsize 53
		generation 138 type 1 (regular)
		extent data disk byte 164798357504 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 85 key (409 INODE_ITEM 0) itemoff 7124 itemsize 160
		generation 137 transid 137 size 262144 nbytes 262144
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743507.817029532 (2018-02-15 17:11:47)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 86 key (409 EXTENT_DATA 0) itemoff 7071 itemsize 53
		generation 137 type 1 (regular)
		extent data disk byte 163291897856 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 87 key (410 INODE_ITEM 0) itemoff 6911 itemsize 160
		generation 1448951 transid 1448951 size 262144 nbytes 6553600
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 25 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699242.6790619 (2020-07-13 21:00:42)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 88 key (410 EXTENT_DATA 0) itemoff 6858 itemsize 53
		generation 1448951 type 1 (regular)
		extent data disk byte 385906823168 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 89 key (411 INODE_ITEM 0) itemoff 6698 itemsize 160
		generation 140 transid 140 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743606.123691212 (2018-02-15 17:13:26)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 90 key (411 EXTENT_DATA 0) itemoff 6645 itemsize 53
		generation 140 type 1 (regular)
		extent data disk byte 166995783680 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 91 key (412 INODE_ITEM 0) itemoff 6485 itemsize 160
		generation 141 transid 141 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743640.935217321 (2018-02-15 17:14:00)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 92 key (412 EXTENT_DATA 0) itemoff 6432 itemsize 53
		generation 141 type 1 (regular)
		extent data disk byte 169143394304 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 93 key (413 INODE_ITEM 0) itemoff 6272 itemsize 160
		generation 1448963 transid 1448963 size 262144 nbytes 10485760
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 40 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699593.240342546 (2020-07-13 21:06:33)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 94 key (413 EXTENT_DATA 0) itemoff 6219 itemsize 53
		generation 1448963 type 1 (regular)
		extent data disk byte 385925726208 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 95 key (414 INODE_ITEM 0) itemoff 6059 itemsize 160
		generation 142 transid 142 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743671.654799143 (2018-02-15 17:14:31)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 96 key (414 EXTENT_DATA 0) itemoff 6006 itemsize 53
		generation 142 type 1 (regular)
		extent data disk byte 171165818880 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 97 key (415 INODE_ITEM 0) itemoff 5846 itemsize 160
		generation 143 transid 143 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743704.426353042 (2018-02-15 17:15:04)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 98 key (415 EXTENT_DATA 0) itemoff 5793 itemsize 53
		generation 143 type 1 (regular)
		extent data disk byte 171290898432 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 99 key (416 INODE_ITEM 0) itemoff 5633 itemsize 160
		generation 143 transid 143 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743704.426353042 (2018-02-15 17:15:04)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 100 key (416 EXTENT_DATA 0) itemoff 5580 itemsize 53
		generation 143 type 1 (regular)
		extent data disk byte 172363669504 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 101 key (417 INODE_ITEM 0) itemoff 5420 itemsize 160
		generation 1448949 transid 1448949 size 262144 nbytes 20447232
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 78 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699168.280995440 (2020-07-13 20:59:28)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 102 key (417 EXTENT_DATA 0) itemoff 5367 itemsize 53
		generation 1448949 type 1 (regular)
		extent data disk byte 385892741120 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 103 key (418 INODE_ITEM 0) itemoff 5207 itemsize 160
		generation 145 transid 145 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743765.865516732 (2018-02-15 17:16:05)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 104 key (418 EXTENT_DATA 0) itemoff 5154 itemsize 53
		generation 145 type 1 (regular)
		extent data disk byte 175585861632 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 105 key (419 INODE_ITEM 0) itemoff 4994 itemsize 160
		generation 144 transid 144 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743735.145934883 (2018-02-15 17:15:35)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 106 key (419 EXTENT_DATA 0) itemoff 4941 itemsize 53
		generation 144 type 1 (regular)
		extent data disk byte 174177488896 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 107 key (420 INODE_ITEM 0) itemoff 4781 itemsize 160
		generation 146 transid 146 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743796.589098537 (2018-02-15 17:16:36)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 108 key (420 EXTENT_DATA 0) itemoff 4728 itemsize 53
		generation 146 type 1 (regular)
		extent data disk byte 177198120960 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 109 key (421 INODE_ITEM 0) itemoff 4568 itemsize 160
		generation 1448965 transid 1448965 size 262144 nbytes 7340032
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 28 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699675.185917664 (2020-07-13 21:07:55)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 110 key (421 EXTENT_DATA 0) itemoff 4515 itemsize 53
		generation 1448965 type 1 (regular)
		extent data disk byte 385927106560 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 111 key (422 INODE_ITEM 0) itemoff 4355 itemsize 160
		generation 147 transid 147 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743829.348652638 (2018-02-15 17:17:09)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 112 key (422 EXTENT_DATA 0) itemoff 4302 itemsize 53
		generation 147 type 1 (regular)
		extent data disk byte 178797867008 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 113 key (423 INODE_ITEM 0) itemoff 4142 itemsize 160
		generation 148 transid 148 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743862.124206531 (2018-02-15 17:17:42)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 114 key (423 EXTENT_DATA 0) itemoff 4089 itemsize 53
		generation 148 type 1 (regular)
		extent data disk byte 179880357888 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 115 key (424 INODE_ITEM 0) itemoff 3929 itemsize 160
		generation 1591706 transid 1591706 size 262144 nbytes 91750400
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 350 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1643596316.171356577 (2022-01-30 18:31:56)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 116 key (424 EXTENT_DATA 0) itemoff 3876 itemsize 53
		generation 1591706 type 1 (regular)
		extent data disk byte 749647687680 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 117 key (425 INODE_ITEM 0) itemoff 3716 itemsize 160
		generation 150 transid 150 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743927.659314565 (2018-02-15 17:18:47)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 118 key (425 EXTENT_DATA 0) itemoff 3663 itemsize 53
		generation 150 type 1 (regular)
		extent data disk byte 183102033920 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 119 key (426 INODE_ITEM 0) itemoff 3503 itemsize 160
		generation 1453661 transid 1453661 size 262144 nbytes 6815744
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 26 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594926629.456877205 (2020-07-16 12:10:29)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 120 key (426 EXTENT_DATA 0) itemoff 3450 itemsize 53
		generation 1453661 type 1 (regular)
		extent data disk byte 747309912064 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 121 key (427 INODE_ITEM 0) itemoff 3290 itemsize 160
		generation 151 transid 151 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743960.426868596 (2018-02-15 17:19:20)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 122 key (427 EXTENT_DATA 0) itemoff 3237 itemsize 53
		generation 151 type 1 (regular)
		extent data disk byte 185019625472 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
leaf 11970963865600 items 123 free space 55 generation 1449566 owner ROOT_TREE
leaf 11970963865600 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
	item 0 key (428 INODE_ITEM 0) itemoff 16123 itemsize 160
		generation 152 transid 152 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743991.142450563 (2018-02-15 17:19:51)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 1 key (428 EXTENT_DATA 0) itemoff 16070 itemsize 53
		generation 152 type 1 (regular)
		extent data disk byte 185249386496 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 2 key (429 INODE_ITEM 0) itemoff 15910 itemsize 160
		generation 152 transid 152 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518743991.142450563 (2018-02-15 17:19:51)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 3 key (429 EXTENT_DATA 0) itemoff 15857 itemsize 53
		generation 152 type 1 (regular)
		extent data disk byte 186323259392 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 4 key (430 INODE_ITEM 0) itemoff 15697 itemsize 160
		generation 1448949 transid 1448949 size 262144 nbytes 7077888
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 27 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699168.280995440 (2020-07-13 20:59:28)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 5 key (430 EXTENT_DATA 0) itemoff 15644 itemsize 53
		generation 1448949 type 1 (regular)
		extent data disk byte 385895763968 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 6 key (431 INODE_ITEM 0) itemoff 15484 itemsize 160
		generation 153 transid 153 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744025.957976741 (2018-02-15 17:20:25)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 7 key (431 EXTENT_DATA 0) itemoff 15431 itemsize 53
		generation 153 type 1 (regular)
		extent data disk byte 188171296768 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 8 key (432 INODE_ITEM 0) itemoff 15271 itemsize 160
		generation 155 transid 155 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744089.449112687 (2018-02-15 17:21:29)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 9 key (432 EXTENT_DATA 0) itemoff 15218 itemsize 53
		generation 155 type 1 (regular)
		extent data disk byte 190618001408 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 10 key (433 INODE_ITEM 0) itemoff 15058 itemsize 160
		generation 155 transid 155 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744089.449112687 (2018-02-15 17:21:29)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 11 key (433 EXTENT_DATA 0) itemoff 15005 itemsize 53
		generation 155 type 1 (regular)
		extent data disk byte 190618263552 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 12 key (434 INODE_ITEM 0) itemoff 14845 itemsize 160
		generation 155 transid 155 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744089.449112687 (2018-02-15 17:21:29)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 13 key (434 EXTENT_DATA 0) itemoff 14792 itemsize 53
		generation 155 type 1 (regular)
		extent data disk byte 191040020480 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 14 key (435 INODE_ITEM 0) itemoff 14632 itemsize 160
		generation 1448966 transid 1448966 size 262144 nbytes 5505024
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 21 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699711.4859189 (2020-07-13 21:08:31)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 15 key (435 EXTENT_DATA 0) itemoff 14579 itemsize 53
		generation 1448966 type 1 (regular)
		extent data disk byte 385915817984 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 16 key (436 INODE_ITEM 0) itemoff 14419 itemsize 160
		generation 158 transid 158 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744187.751774948 (2018-02-15 17:23:07)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 17 key (436 EXTENT_DATA 0) itemoff 14366 itemsize 53
		generation 158 type 1 (regular)
		extent data disk byte 195433910272 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 18 key (437 INODE_ITEM 0) itemoff 14206 itemsize 160
		generation 158 transid 158 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744187.751774948 (2018-02-15 17:23:07)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 19 key (437 EXTENT_DATA 0) itemoff 14153 itemsize 53
		generation 158 type 1 (regular)
		extent data disk byte 195434172416 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 20 key (438 INODE_ITEM 0) itemoff 13993 itemsize 160
		generation 1448965 transid 1448965 size 262144 nbytes 6291456
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 24 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699675.185917664 (2020-07-13 21:07:55)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 21 key (438 EXTENT_DATA 0) itemoff 13940 itemsize 53
		generation 1448965 type 1 (regular)
		extent data disk byte 385927368704 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 22 key (439 INODE_ITEM 0) itemoff 13780 itemsize 160
		generation 160 transid 160 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744251.242910981 (2018-02-15 17:24:11)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 23 key (439 EXTENT_DATA 0) itemoff 13727 itemsize 53
		generation 160 type 1 (regular)
		extent data disk byte 198565240832 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 24 key (440 INODE_ITEM 0) itemoff 13567 itemsize 160
		generation 160 transid 160 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744251.242910981 (2018-02-15 17:24:11)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 25 key (440 EXTENT_DATA 0) itemoff 13514 itemsize 53
		generation 160 type 1 (regular)
		extent data disk byte 198565502976 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 26 key (441 INODE_ITEM 0) itemoff 13354 itemsize 160
		generation 1448966 transid 1448966 size 262144 nbytes 7602176
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 29 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699711.16858834 (2020-07-13 21:08:31)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 27 key (441 EXTENT_DATA 0) itemoff 13301 itemsize 53
		generation 1448966 type 1 (regular)
		extent data disk byte 385924657152 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 28 key (442 INODE_ITEM 0) itemoff 13141 itemsize 160
		generation 160 transid 160 size 262144 nbytes 262144
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744251.242910981 (2018-02-15 17:24:11)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 29 key (442 EXTENT_DATA 0) itemoff 13088 itemsize 53
		generation 160 type 1 (regular)
		extent data disk byte 198566027264 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 30 key (443 INODE_ITEM 0) itemoff 12928 itemsize 160
		generation 163 transid 163 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744349.541573431 (2018-02-15 17:25:49)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 31 key (443 EXTENT_DATA 0) itemoff 12875 itemsize 53
		generation 163 type 1 (regular)
		extent data disk byte 203380895744 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 32 key (444 INODE_ITEM 0) itemoff 12715 itemsize 160
		generation 1449566 transid 1449566 size 262144 nbytes 5505024
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 21 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594721866.262712109 (2020-07-14 03:17:46)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 33 key (444 EXTENT_DATA 0) itemoff 12662 itemsize 53
		generation 1449566 type 1 (regular)
		extent data disk byte 386341617664 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 34 key (445 INODE_ITEM 0) itemoff 12502 itemsize 160
		generation 162 transid 162 size 262144 nbytes 262144
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744314.734047048 (2018-02-15 17:25:14)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 35 key (445 EXTENT_DATA 0) itemoff 12449 itemsize 53
		generation 162 type 1 (regular)
		extent data disk byte 201780236288 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 36 key (446 INODE_ITEM 0) itemoff 12289 itemsize 160
		generation 164 transid 164 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744382.309127580 (2018-02-15 17:26:22)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 37 key (446 EXTENT_DATA 0) itemoff 12236 itemsize 53
		generation 164 type 1 (regular)
		extent data disk byte 204576649216 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 38 key (447 INODE_ITEM 0) itemoff 12076 itemsize 160
		generation 1448971 transid 1448971 size 262144 nbytes 6029312
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 23 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699900.442400193 (2020-07-13 21:11:40)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 39 key (447 EXTENT_DATA 0) itemoff 12023 itemsize 53
		generation 1448971 type 1 (regular)
		extent data disk byte 385932079104 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 40 key (448 INODE_ITEM 0) itemoff 11863 itemsize 160
		generation 166 transid 166 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744447.852235796 (2018-02-15 17:27:27)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 41 key (448 EXTENT_DATA 0) itemoff 11810 itemsize 53
		generation 166 type 1 (regular)
		extent data disk byte 208198201344 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 42 key (449 INODE_ITEM 0) itemoff 11650 itemsize 160
		generation 166 transid 166 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744447.852235796 (2018-02-15 17:27:27)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 43 key (449 EXTENT_DATA 0) itemoff 11597 itemsize 53
		generation 166 type 1 (regular)
		extent data disk byte 208198463488 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 44 key (450 INODE_ITEM 0) itemoff 11437 itemsize 160
		generation 1519 transid 1519 size 262144 nbytes 7864320
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 30 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518802433.118441774 (2018-02-16 09:33:53)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 45 key (450 EXTENT_DATA 0) itemoff 11384 itemsize 53
		generation 1519 type 1 (regular)
		extent data disk byte 1145710120960 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 46 key (451 INODE_ITEM 0) itemoff 11224 itemsize 160
		generation 168 transid 168 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744513.387344154 (2018-02-15 17:28:33)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 47 key (451 EXTENT_DATA 0) itemoff 11171 itemsize 53
		generation 168 type 1 (regular)
		extent data disk byte 211231223808 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 48 key (452 INODE_ITEM 0) itemoff 11011 itemsize 160
		generation 168 transid 168 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744513.387344154 (2018-02-15 17:28:33)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 49 key (452 EXTENT_DATA 0) itemoff 10958 itemsize 53
		generation 168 type 1 (regular)
		extent data disk byte 211231485952 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 50 key (453 INODE_ITEM 0) itemoff 10798 itemsize 160
		generation 1498 transid 1498 size 262144 nbytes 6553600
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 25 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518801363.816988131 (2018-02-16 09:16:03)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 51 key (453 EXTENT_DATA 0) itemoff 10745 itemsize 53
		generation 1498 type 1 (regular)
		extent data disk byte 946510630912 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 52 key (454 INODE_ITEM 0) itemoff 10585 itemsize 160
		generation 170 transid 170 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744574.934476684 (2018-02-15 17:29:34)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 53 key (454 EXTENT_DATA 0) itemoff 10532 itemsize 53
		generation 170 type 1 (regular)
		extent data disk byte 214238515200 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 54 key (455 INODE_ITEM 0) itemoff 10372 itemsize 160
		generation 170 transid 170 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744574.934476684 (2018-02-15 17:29:34)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 55 key (455 EXTENT_DATA 0) itemoff 10319 itemsize 53
		generation 170 type 1 (regular)
		extent data disk byte 214238777344 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 56 key (456 INODE_ITEM 0) itemoff 10159 itemsize 160
		generation 2685 transid 2685 size 262144 nbytes 17563648
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 67 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518855339.205406132 (2018-02-17 00:15:39)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 57 key (456 EXTENT_DATA 0) itemoff 10106 itemsize 53
		generation 2685 type 1 (regular)
		extent data disk byte 1105982980096 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 58 key (457 INODE_ITEM 0) itemoff 9946 itemsize 160
		generation 172 transid 172 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744638.509573221 (2018-02-15 17:30:38)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 59 key (457 EXTENT_DATA 0) itemoff 9893 itemsize 53
		generation 172 type 1 (regular)
		extent data disk byte 217410400256 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 60 key (458 INODE_ITEM 0) itemoff 9733 itemsize 160
		generation 173 transid 173 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744671.85110480 (2018-02-15 17:31:11)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 61 key (458 EXTENT_DATA 0) itemoff 9680 itemsize 53
		generation 173 type 1 (regular)
		extent data disk byte 218879705088 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 62 key (459 INODE_ITEM 0) itemoff 9520 itemsize 160
		generation 173 transid 173 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744671.85110480 (2018-02-15 17:31:11)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 63 key (459 EXTENT_DATA 0) itemoff 9467 itemsize 53
		generation 173 type 1 (regular)
		extent data disk byte 218880229376 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 64 key (460 INODE_ITEM 0) itemoff 9307 itemsize 160
		generation 1448949 transid 1448949 size 262144 nbytes 15466496
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 59 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699168.280995440 (2020-07-13 20:59:28)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 65 key (460 EXTENT_DATA 0) itemoff 9254 itemsize 53
		generation 1448949 type 1 (regular)
		extent data disk byte 385898123264 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 66 key (461 INODE_ITEM 0) itemoff 9094 itemsize 160
		generation 173 transid 173 size 262144 nbytes 262144
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744671.85110480 (2018-02-15 17:31:11)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 67 key (461 EXTENT_DATA 0) itemoff 9041 itemsize 53
		generation 173 type 1 (regular)
		extent data disk byte 218880491520 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 68 key (462 INODE_ITEM 0) itemoff 8881 itemsize 160
		generation 175 transid 175 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744732.520238131 (2018-02-15 17:32:12)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 69 key (462 EXTENT_DATA 0) itemoff 8828 itemsize 53
		generation 175 type 1 (regular)
		extent data disk byte 221741686784 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 70 key (463 INODE_ITEM 0) itemoff 8668 itemsize 160
		generation 1448971 transid 1448971 size 262144 nbytes 56623104
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 216 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699900.442400193 (2020-07-13 21:11:40)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 71 key (463 EXTENT_DATA 0) itemoff 8615 itemsize 53
		generation 1448971 type 1 (regular)
		extent data disk byte 385932341248 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 72 key (464 INODE_ITEM 0) itemoff 8455 itemsize 160
		generation 179 transid 179 size 262144 nbytes 1310720
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744859.842431625 (2018-02-15 17:34:19)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 73 key (464 EXTENT_DATA 0) itemoff 8402 itemsize 53
		generation 179 type 1 (regular)
		extent data disk byte 224975945728 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 74 key (465 INODE_ITEM 0) itemoff 8242 itemsize 160
		generation 179 transid 179 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744859.842431625 (2018-02-15 17:34:19)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 75 key (465 EXTENT_DATA 0) itemoff 8189 itemsize 53
		generation 179 type 1 (regular)
		extent data disk byte 224976207872 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 76 key (466 INODE_ITEM 0) itemoff 8029 itemsize 160
		generation 180 transid 180 size 262144 nbytes 1310720
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744892.261971940 (2018-02-15 17:34:52)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 77 key (466 EXTENT_DATA 0) itemoff 7976 itemsize 53
		generation 180 type 1 (regular)
		extent data disk byte 226577911808 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 78 key (467 INODE_ITEM 0) itemoff 7816 itemsize 160
		generation 181 transid 181 size 262144 nbytes 1310720
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744925.33507385 (2018-02-15 17:35:25)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 79 key (467 EXTENT_DATA 0) itemoff 7763 itemsize 53
		generation 181 type 1 (regular)
		extent data disk byte 228198100992 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 80 key (468 INODE_ITEM 0) itemoff 7603 itemsize 160
		generation 181 transid 181 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518744925.33507385 (2018-02-15 17:35:25)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 81 key (468 EXTENT_DATA 0) itemoff 7550 itemsize 53
		generation 181 type 1 (regular)
		extent data disk byte 228198363136 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 82 key (469 INODE_ITEM 0) itemoff 7390 itemsize 160
		generation 1448969 transid 1448969 size 262144 nbytes 45613056
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 174 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699831.833295032 (2020-07-13 21:10:31)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 83 key (469 EXTENT_DATA 0) itemoff 7337 itemsize 53
		generation 1448969 type 1 (regular)
		extent data disk byte 385929334784 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 84 key (470 INODE_ITEM 0) itemoff 7177 itemsize 160
		generation 186 transid 186 size 262144 nbytes 2097152
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 8 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745090.967119882 (2018-02-15 17:38:10)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 85 key (470 EXTENT_DATA 0) itemoff 7124 itemsize 53
		generation 186 type 1 (regular)
		extent data disk byte 232493420544 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 86 key (471 INODE_ITEM 0) itemoff 6964 itemsize 160
		generation 190 transid 190 size 262144 nbytes 2883584
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745219.945264147 (2018-02-15 17:40:19)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 87 key (471 EXTENT_DATA 0) itemoff 6911 itemsize 53
		generation 190 type 1 (regular)
		extent data disk byte 233566826496 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 88 key (472 INODE_ITEM 0) itemoff 6751 itemsize 160
		generation 1448949 transid 1448949 size 262144 nbytes 11272192
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 43 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699168.280995440 (2020-07-13 20:59:28)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 89 key (472 EXTENT_DATA 0) itemoff 6698 itemsize 53
		generation 1448949 type 1 (regular)
		extent data disk byte 385899433984 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 90 key (473 INODE_ITEM 0) itemoff 6538 itemsize 160
		generation 189 transid 189 size 262144 nbytes 2097152
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 8 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745189.229705836 (2018-02-15 17:39:49)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 91 key (473 EXTENT_DATA 0) itemoff 6485 itemsize 53
		generation 189 type 1 (regular)
		extent data disk byte 233567481856 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 92 key (474 INODE_ITEM 0) itemoff 6325 itemsize 160
		generation 192 transid 192 size 262144 nbytes 3145728
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745287.536292726 (2018-02-15 17:41:27)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 93 key (474 EXTENT_DATA 0) itemoff 6272 itemsize 53
		generation 192 type 1 (regular)
		extent data disk byte 235713855488 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 94 key (475 INODE_ITEM 0) itemoff 6112 itemsize 160
		generation 1448952 transid 1448952 size 262144 nbytes 8650752
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 33 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699261.674203156 (2020-07-13 21:01:01)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 95 key (475 EXTENT_DATA 0) itemoff 6059 itemsize 53
		generation 1448952 type 1 (regular)
		extent data disk byte 385905856512 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 96 key (476 INODE_ITEM 0) itemoff 5899 itemsize 160
		generation 202 transid 202 size 262144 nbytes 3407872
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 13 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745621.355491718 (2018-02-15 17:47:01)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 97 key (476 EXTENT_DATA 0) itemoff 5846 itemsize 53
		generation 202 type 1 (regular)
		extent data disk byte 247526436864 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 98 key (477 INODE_ITEM 0) itemoff 5686 itemsize 160
		generation 196 transid 196 size 262144 nbytes 3145728
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745416.550440527 (2018-02-15 17:43:36)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 99 key (477 EXTENT_DATA 0) itemoff 5633 itemsize 53
		generation 196 type 1 (regular)
		extent data disk byte 241084006400 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 100 key (478 INODE_ITEM 0) itemoff 5473 itemsize 160
		generation 197 transid 197 size 262144 nbytes 2621440
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 10 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745451.389940795 (2018-02-15 17:44:11)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 101 key (478 EXTENT_DATA 0) itemoff 5420 itemsize 53
		generation 197 type 1 (regular)
		extent data disk byte 240008871936 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 102 key (479 INODE_ITEM 0) itemoff 5260 itemsize 160
		generation 196 transid 196 size 262144 nbytes 1572864
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 6 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745416.550440527 (2018-02-15 17:43:36)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 103 key (479 EXTENT_DATA 0) itemoff 5207 itemsize 53
		generation 196 type 1 (regular)
		extent data disk byte 242157457408 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 104 key (480 INODE_ITEM 0) itemoff 5047 itemsize 160
		generation 2370 transid 2370 size 262144 nbytes 22544384
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 86 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518841689.246621552 (2018-02-16 20:28:09)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 105 key (480 EXTENT_DATA 0) itemoff 4994 itemsize 53
		generation 2370 type 1 (regular)
		extent data disk byte 1247717027840 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 106 key (481 INODE_ITEM 0) itemoff 4834 itemsize 160
		generation 1519 transid 1519 size 262144 nbytes 12845056
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 49 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518802433.118441774 (2018-02-16 09:33:53)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 107 key (481 EXTENT_DATA 0) itemoff 4781 itemsize 53
		generation 1519 type 1 (regular)
		extent data disk byte 1145711169536 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 108 key (482 INODE_ITEM 0) itemoff 4621 itemsize 160
		generation 199 transid 199 size 262144 nbytes 2883584
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745520.996942914 (2018-02-15 17:45:20)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 109 key (482 EXTENT_DATA 0) itemoff 4568 itemsize 53
		generation 199 type 1 (regular)
		extent data disk byte 248600330240 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 110 key (483 INODE_ITEM 0) itemoff 4408 itemsize 160
		generation 201 transid 201 size 262144 nbytes 3145728
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745588.671966038 (2018-02-15 17:46:28)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 111 key (483 EXTENT_DATA 0) itemoff 4355 itemsize 53
		generation 201 type 1 (regular)
		extent data disk byte 256116416512 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 112 key (484 INODE_ITEM 0) itemoff 4195 itemsize 160
		generation 201 transid 201 size 262144 nbytes 2621440
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 10 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745588.671966038 (2018-02-15 17:46:28)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 113 key (484 EXTENT_DATA 0) itemoff 4142 itemsize 53
		generation 201 type 1 (regular)
		extent data disk byte 257190248448 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 114 key (485 INODE_ITEM 0) itemoff 3982 itemsize 160
		generation 202 transid 202 size 262144 nbytes 3145728
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745621.355491718 (2018-02-15 17:47:01)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 115 key (485 EXTENT_DATA 0) itemoff 3929 itemsize 53
		generation 202 type 1 (regular)
		extent data disk byte 251821535232 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 116 key (486 INODE_ITEM 0) itemoff 3769 itemsize 160
		generation 202 transid 202 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745621.355491718 (2018-02-15 17:47:01)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 117 key (486 EXTENT_DATA 0) itemoff 3716 itemsize 53
		generation 202 type 1 (regular)
		extent data disk byte 248599805952 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 118 key (487 INODE_ITEM 0) itemoff 3556 itemsize 160
		generation 202 transid 202 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745621.355491718 (2018-02-15 17:47:01)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 119 key (487 EXTENT_DATA 0) itemoff 3503 itemsize 53
		generation 202 type 1 (regular)
		extent data disk byte 252894867456 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 120 key (488 INODE_ITEM 0) itemoff 3343 itemsize 160
		generation 204 transid 204 size 262144 nbytes 3407872
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 13 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745688.938511563 (2018-02-15 17:48:08)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 121 key (488 EXTENT_DATA 0) itemoff 3290 itemsize 53
		generation 204 type 1 (regular)
		extent data disk byte 269818867712 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 122 key (489 INODE_ITEM 0) itemoff 3130 itemsize 160
		generation 204 transid 204 size 262144 nbytes 3407872
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 13 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518745688.938511563 (2018-02-15 17:48:08)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
parent transid verify failed on 13576971354112 wanted 1601940 found 1602224
parent transid verify failed on 13576971354112 wanted 1601940 found 1602224
parent transid verify failed on 13576971354112 wanted 1601940 found 1602224
Ignoring transid failure
leaf 13576971354112 items 204 free space 4388 generation 1602224 owner EXTENT_TREE
leaf 13576971354112 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
	item 0 key (13577814360064 METADATA_ITEM 0) itemoff 16250 itemsize 33
		refs 1 gen 1365916 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 1 key (13577814376448 METADATA_ITEM 0) itemoff 16217 itemsize 33
		refs 1 gen 1365916 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 2 key (13577814392832 METADATA_ITEM 0) itemoff 16184 itemsize 33
		refs 1 gen 1365916 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 3 key (13577814409216 METADATA_ITEM 0) itemoff 16151 itemsize 33
		refs 1 gen 1365916 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 4 key (13577814425600 METADATA_ITEM 0) itemoff 16118 itemsize 33
		refs 1 gen 1365916 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 5 key (13577814441984 METADATA_ITEM 0) itemoff 16085 itemsize 33
		refs 1 gen 1365916 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 6 key (13577814458368 METADATA_ITEM 0) itemoff 16052 itemsize 33
		refs 1 gen 1365916 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 7 key (13577814474752 METADATA_ITEM 0) itemoff 16019 itemsize 33
		refs 1 gen 1397297 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 8 key (13577814491136 METADATA_ITEM 0) itemoff 15986 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 9 key (13577814507520 METADATA_ITEM 0) itemoff 15953 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 10 key (13577814523904 METADATA_ITEM 0) itemoff 15920 itemsize 33
		refs 1 gen 1524027 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 11 key (13577814556672 METADATA_ITEM 0) itemoff 15887 itemsize 33
		refs 1 gen 1510369 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 12 key (13577814589440 METADATA_ITEM 0) itemoff 15854 itemsize 33
		refs 1 gen 1371035 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 13 key (13577814605824 METADATA_ITEM 0) itemoff 15821 itemsize 33
		refs 1 gen 1508897 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 14 key (13577814638592 METADATA_ITEM 0) itemoff 15788 itemsize 33
		refs 1 gen 1524031 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 15 key (13577814654976 METADATA_ITEM 0) itemoff 15755 itemsize 33
		refs 1 gen 1397297 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 16 key (13577814671360 METADATA_ITEM 0) itemoff 15722 itemsize 33
		refs 1 gen 1377633 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 17 key (13577814687744 METADATA_ITEM 0) itemoff 15689 itemsize 33
		refs 1 gen 1397297 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 18 key (13577814704128 METADATA_ITEM 0) itemoff 15656 itemsize 33
		refs 1 gen 1488520 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 19 key (13577814736896 METADATA_ITEM 0) itemoff 15623 itemsize 33
		refs 1 gen 1397297 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 20 key (13577814753280 METADATA_ITEM 0) itemoff 15590 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 21 key (13577814769664 METADATA_ITEM 0) itemoff 15557 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 22 key (13577814786048 METADATA_ITEM 0) itemoff 15524 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 23 key (13577814802432 METADATA_ITEM 0) itemoff 15491 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 24 key (13577814818816 METADATA_ITEM 0) itemoff 15458 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 25 key (13577814835200 METADATA_ITEM 0) itemoff 15425 itemsize 33
		refs 1 gen 1524026 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 26 key (13577814851584 METADATA_ITEM 0) itemoff 15392 itemsize 33
		refs 1 gen 1174448 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 27 key (13577814867968 METADATA_ITEM 0) itemoff 15359 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 28 key (13577814884352 METADATA_ITEM 0) itemoff 15326 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 29 key (13577814900736 METADATA_ITEM 0) itemoff 15293 itemsize 33
		refs 1 gen 1397297 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 30 key (13577814933504 METADATA_ITEM 0) itemoff 15260 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 31 key (13577814949888 METADATA_ITEM 0) itemoff 15227 itemsize 33
		refs 1 gen 1509021 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 32 key (13577814966272 METADATA_ITEM 0) itemoff 15194 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 33 key (13577814982656 METADATA_ITEM 0) itemoff 15161 itemsize 33
		refs 1 gen 1538429 flags TREE_BLOCK
		tree block skinny level 0
		shared block backref parent 15645094871040
	item 34 key (13577814999040 METADATA_ITEM 0) itemoff 15128 itemsize 33
		refs 1 gen 1524007 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 35 key (13577815015424 METADATA_ITEM 0) itemoff 15095 itemsize 33
		refs 1 gen 1482916 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 36 key (13577815031808 METADATA_ITEM 1) itemoff 15062 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 1
		tree block backref root CSUM_TREE
	item 37 key (13577815048192 METADATA_ITEM 0) itemoff 15029 itemsize 33
		refs 1 gen 1397299 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 38 key (13577815064576 METADATA_ITEM 0) itemoff 14996 itemsize 33
		refs 1 gen 1441746 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 39 key (13577815080960 METADATA_ITEM 0) itemoff 14963 itemsize 33
		refs 1 gen 1353162 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 40 key (13577815097344 METADATA_ITEM 0) itemoff 14930 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 41 key (13577815113728 METADATA_ITEM 0) itemoff 14897 itemsize 33
		refs 1 gen 1524007 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 42 key (13577815130112 METADATA_ITEM 0) itemoff 14864 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 43 key (13577815146496 METADATA_ITEM 0) itemoff 14831 itemsize 33
		refs 1 gen 1170837 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 44 key (13577815162880 METADATA_ITEM 0) itemoff 14798 itemsize 33
		refs 1 gen 1250069 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 45 key (13577815179264 METADATA_ITEM 0) itemoff 14765 itemsize 33
		refs 1 gen 1482916 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 46 key (13577815195648 METADATA_ITEM 0) itemoff 14732 itemsize 33
		refs 1 gen 1250069 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 47 key (13577815212032 METADATA_ITEM 0) itemoff 14699 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 48 key (13577815228416 METADATA_ITEM 0) itemoff 14666 itemsize 33
		refs 1 gen 1509021 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 49 key (13577815244800 METADATA_ITEM 0) itemoff 14633 itemsize 33
		refs 1 gen 1173729 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 50 key (13577815261184 METADATA_ITEM 0) itemoff 14600 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 51 key (13577815277568 METADATA_ITEM 0) itemoff 14567 itemsize 33
		refs 1 gen 1525997 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 52 key (13577815293952 METADATA_ITEM 0) itemoff 14534 itemsize 33
		refs 1 gen 1441746 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 53 key (13577815310336 METADATA_ITEM 0) itemoff 14501 itemsize 33
		refs 1 gen 1524033 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 54 key (13577815326720 METADATA_ITEM 0) itemoff 14468 itemsize 33
		refs 1 gen 1399581 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 55 key (13577815343104 METADATA_ITEM 0) itemoff 14435 itemsize 33
		refs 1 gen 1462080 flags TREE_BLOCK|FULL_BACKREF
		tree block skinny level 0
		shared block backref parent 10679216750592
	item 56 key (13577815359488 METADATA_ITEM 0) itemoff 14402 itemsize 33
		refs 1 gen 1509021 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 57 key (13577815375872 METADATA_ITEM 0) itemoff 14369 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 58 key (13577815392256 METADATA_ITEM 0) itemoff 14336 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 59 key (13577815408640 METADATA_ITEM 0) itemoff 14303 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 60 key (13577815425024 METADATA_ITEM 0) itemoff 14270 itemsize 33
		refs 1 gen 1509130 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 61 key (13577815441408 METADATA_ITEM 0) itemoff 14237 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 62 key (13577815457792 METADATA_ITEM 0) itemoff 14204 itemsize 33
		refs 1 gen 1397299 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 63 key (13577815474176 METADATA_ITEM 0) itemoff 14171 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 64 key (13577815490560 METADATA_ITEM 0) itemoff 14138 itemsize 33
		refs 1 gen 1371037 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 65 key (13577815506944 METADATA_ITEM 0) itemoff 14105 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 66 key (13577815523328 METADATA_ITEM 0) itemoff 14072 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 67 key (13577815539712 METADATA_ITEM 0) itemoff 14039 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 68 key (13577815556096 METADATA_ITEM 0) itemoff 14006 itemsize 33
		refs 1 gen 1390120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 69 key (13577815572480 METADATA_ITEM 0) itemoff 13973 itemsize 33
		refs 1 gen 1536322 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 70 key (13577815588864 METADATA_ITEM 0) itemoff 13940 itemsize 33
		refs 1 gen 1397299 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 71 key (13577815605248 METADATA_ITEM 0) itemoff 13907 itemsize 33
		refs 1 gen 1510274 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 72 key (13577815621632 METADATA_ITEM 0) itemoff 13874 itemsize 33
		refs 1 gen 1371036 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 73 key (13577815638016 METADATA_ITEM 0) itemoff 13841 itemsize 33
		refs 1 gen 1536322 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 74 key (13577815654400 METADATA_ITEM 0) itemoff 13808 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 75 key (13577815687168 METADATA_ITEM 0) itemoff 13775 itemsize 33
		refs 1 gen 1397299 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 76 key (13577815719936 METADATA_ITEM 0) itemoff 13742 itemsize 33
		refs 1 gen 1522187 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 77 key (13577815736320 METADATA_ITEM 0) itemoff 13709 itemsize 33
		refs 1 gen 1397299 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 78 key (13577815752704 METADATA_ITEM 0) itemoff 13676 itemsize 33
		refs 1 gen 1442268 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root 11221
	item 79 key (13577815769088 METADATA_ITEM 0) itemoff 13643 itemsize 33
		refs 1 gen 1397299 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 80 key (13577815785472 METADATA_ITEM 0) itemoff 13610 itemsize 33
		refs 1 gen 1453429 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root FS_TREE
	item 81 key (13577815801856 METADATA_ITEM 0) itemoff 13577 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 82 key (13577815818240 METADATA_ITEM 0) itemoff 13544 itemsize 33
		refs 1 gen 1397299 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 83 key (13577815834624 METADATA_ITEM 0) itemoff 13511 itemsize 33
		refs 1 gen 1377634 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 84 key (13577815851008 METADATA_ITEM 0) itemoff 13478 itemsize 33
		refs 1 gen 1524033 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 85 key (13577815867392 METADATA_ITEM 0) itemoff 13445 itemsize 33
		refs 1 gen 1397299 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 86 key (13577815883776 METADATA_ITEM 0) itemoff 13412 itemsize 33
		refs 1 gen 1397299 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 87 key (13577815900160 METADATA_ITEM 0) itemoff 13379 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 88 key (13577815932928 METADATA_ITEM 0) itemoff 13346 itemsize 33
		refs 1 gen 1509021 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root 11288
	item 89 key (13577815949312 METADATA_ITEM 0) itemoff 13313 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 90 key (13577815965696 METADATA_ITEM 0) itemoff 13280 itemsize 33
		refs 1 gen 1443708 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 91 key (13577815998464 METADATA_ITEM 0) itemoff 13247 itemsize 33
		refs 1 gen 1534021 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 92 key (13577816014848 METADATA_ITEM 0) itemoff 13214 itemsize 33
		refs 1 gen 1534021 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 93 key (13577816031232 METADATA_ITEM 0) itemoff 13118 itemsize 96
		refs 8 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root 11288
		shared block backref parent 15645306437632
		shared block backref parent 11822225473536
		shared block backref parent 4866918383616
		shared block backref parent 4866918072320
		shared block backref parent 4866917679104
		shared block backref parent 4866917367808
		shared block backref parent 4866908192768
	item 94 key (13577816047616 METADATA_ITEM 0) itemoff 13085 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 95 key (13577816064000 METADATA_ITEM 0) itemoff 13052 itemsize 33
		refs 1 gen 1440669 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 96 key (13577816080384 METADATA_ITEM 0) itemoff 13019 itemsize 33
		refs 1 gen 1440669 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 97 key (13577816096768 METADATA_ITEM 0) itemoff 12986 itemsize 33
		refs 1 gen 1440669 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 98 key (13577816113152 METADATA_ITEM 0) itemoff 12953 itemsize 33
		refs 1 gen 1440669 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 99 key (13577816129536 METADATA_ITEM 0) itemoff 12920 itemsize 33
		refs 1 gen 1369803 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 100 key (13577816145920 METADATA_ITEM 0) itemoff 12887 itemsize 33
		refs 1 gen 1399582 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 101 key (13577816162304 METADATA_ITEM 0) itemoff 12854 itemsize 33
		refs 1 gen 1369803 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 102 key (13577816178688 METADATA_ITEM 0) itemoff 12821 itemsize 33
		refs 1 gen 1399582 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 103 key (13577816195072 METADATA_ITEM 0) itemoff 12788 itemsize 33
		refs 1 gen 1365917 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 104 key (13577816211456 METADATA_ITEM 0) itemoff 12755 itemsize 33
		refs 1 gen 1365917 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 105 key (13577816227840 METADATA_ITEM 0) itemoff 12722 itemsize 33
		refs 1 gen 1365917 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 106 key (13577816244224 METADATA_ITEM 0) itemoff 12689 itemsize 33
		refs 1 gen 1365917 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 107 key (13577816260608 METADATA_ITEM 0) itemoff 12656 itemsize 33
		refs 1 gen 1365917 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 108 key (13577816276992 METADATA_ITEM 0) itemoff 12623 itemsize 33
		refs 1 gen 1482917 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 109 key (13577816293376 METADATA_ITEM 0) itemoff 12590 itemsize 33
		refs 1 gen 1522210 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 110 key (13577816309760 METADATA_ITEM 0) itemoff 12557 itemsize 33
		refs 1 gen 1482917 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 111 key (13577816326144 METADATA_ITEM 0) itemoff 12524 itemsize 33
		refs 1 gen 1365917 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 112 key (13577816342528 METADATA_ITEM 0) itemoff 12491 itemsize 33
		refs 1 gen 1369803 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 113 key (13577816358912 METADATA_ITEM 0) itemoff 12458 itemsize 33
		refs 1 gen 1509637 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 114 key (13577816375296 METADATA_ITEM 0) itemoff 12425 itemsize 33
		refs 1 gen 1534021 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 115 key (13577816391680 METADATA_ITEM 0) itemoff 12392 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 116 key (13577816408064 METADATA_ITEM 0) itemoff 12359 itemsize 33
		refs 1 gen 1371038 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 117 key (13577816424448 METADATA_ITEM 0) itemoff 12326 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 118 key (13577816440832 METADATA_ITEM 0) itemoff 12293 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 119 key (13577816457216 METADATA_ITEM 0) itemoff 12260 itemsize 33
		refs 1 gen 1371038 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 120 key (13577816473600 METADATA_ITEM 0) itemoff 12227 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 121 key (13577816489984 METADATA_ITEM 0) itemoff 12194 itemsize 33
		refs 1 gen 1369803 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 122 key (13577816506368 METADATA_ITEM 0) itemoff 12161 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 123 key (13577816522752 METADATA_ITEM 0) itemoff 12128 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 124 key (13577816539136 METADATA_ITEM 0) itemoff 12095 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 125 key (13577816555520 METADATA_ITEM 0) itemoff 12062 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 126 key (13577816571904 METADATA_ITEM 0) itemoff 12029 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 127 key (13577816588288 METADATA_ITEM 0) itemoff 11996 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 128 key (13577816604672 METADATA_ITEM 0) itemoff 11963 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 129 key (13577816621056 METADATA_ITEM 0) itemoff 11930 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 130 key (13577816637440 METADATA_ITEM 0) itemoff 11897 itemsize 33
		refs 1 gen 1369802 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 131 key (13577816653824 METADATA_ITEM 0) itemoff 11864 itemsize 33
		refs 1 gen 1369802 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 132 key (13577816670208 METADATA_ITEM 0) itemoff 11831 itemsize 33
		refs 1 gen 1369802 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 133 key (13577816686592 METADATA_ITEM 0) itemoff 11798 itemsize 33
		refs 1 gen 1369802 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 134 key (13577816702976 METADATA_ITEM 0) itemoff 11765 itemsize 33
		refs 1 gen 1369802 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 135 key (13577816719360 METADATA_ITEM 0) itemoff 11732 itemsize 33
		refs 1 gen 1369802 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 136 key (13577816735744 METADATA_ITEM 0) itemoff 11699 itemsize 33
		refs 1 gen 1500489 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 137 key (13577816752128 METADATA_ITEM 0) itemoff 11666 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 138 key (13577816768512 METADATA_ITEM 0) itemoff 11633 itemsize 33
		refs 1 gen 1369802 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 139 key (13577816784896 METADATA_ITEM 0) itemoff 11600 itemsize 33
		refs 1 gen 1369802 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 140 key (13577816801280 METADATA_ITEM 0) itemoff 11567 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 141 key (13577816817664 METADATA_ITEM 0) itemoff 11534 itemsize 33
		refs 1 gen 1369802 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 142 key (13577816834048 METADATA_ITEM 0) itemoff 11501 itemsize 33
		refs 1 gen 1369802 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 143 key (13577816850432 METADATA_ITEM 0) itemoff 11468 itemsize 33
		refs 1 gen 1397293 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 144 key (13577816866816 METADATA_ITEM 0) itemoff 11435 itemsize 33
		refs 1 gen 1369804 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 145 key (13577816883200 METADATA_ITEM 0) itemoff 11402 itemsize 33
		refs 1 gen 1369804 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 146 key (13577816899584 METADATA_ITEM 0) itemoff 11369 itemsize 33
		refs 1 gen 1369804 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 147 key (13577816915968 METADATA_ITEM 0) itemoff 11336 itemsize 33
		refs 1 gen 1397293 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 148 key (13577816932352 METADATA_ITEM 0) itemoff 11303 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 149 key (13577816948736 METADATA_ITEM 0) itemoff 11270 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 150 key (13577816965120 METADATA_ITEM 0) itemoff 11237 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 151 key (13577816981504 METADATA_ITEM 0) itemoff 11204 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 152 key (13577816997888 METADATA_ITEM 0) itemoff 11171 itemsize 33
		refs 1 gen 1377635 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 153 key (13577817014272 METADATA_ITEM 0) itemoff 11138 itemsize 33
		refs 1 gen 1369804 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 154 key (13577817030656 METADATA_ITEM 0) itemoff 11105 itemsize 33
		refs 1 gen 1369804 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 155 key (13577817047040 METADATA_ITEM 0) itemoff 11072 itemsize 33
		refs 1 gen 1397293 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 156 key (13577817063424 METADATA_ITEM 0) itemoff 11039 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 157 key (13577817079808 METADATA_ITEM 0) itemoff 11006 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 158 key (13577817096192 METADATA_ITEM 0) itemoff 10973 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 159 key (13577817112576 METADATA_ITEM 0) itemoff 10940 itemsize 33
		refs 1 gen 1090952 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 160 key (13577817128960 METADATA_ITEM 0) itemoff 10907 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 161 key (13577817145344 METADATA_ITEM 0) itemoff 10874 itemsize 33
		refs 1 gen 1396849 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 162 key (13577817178112 METADATA_ITEM 0) itemoff 10841 itemsize 33
		refs 1 gen 1443709 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 163 key (13577817194496 METADATA_ITEM 0) itemoff 10808 itemsize 33
		refs 1 gen 1397293 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 164 key (13577817227264 METADATA_ITEM 0) itemoff 10775 itemsize 33
		refs 1 gen 1371039 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 165 key (13577817243648 METADATA_ITEM 0) itemoff 10742 itemsize 33
		refs 1 gen 1397293 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 166 key (13577817260032 METADATA_ITEM 0) itemoff 10709 itemsize 33
		refs 1 gen 1537120 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 167 key (13577817276416 METADATA_ITEM 0) itemoff 10676 itemsize 33
		refs 1 gen 1443708 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 168 key (13577817292800 METADATA_ITEM 0) itemoff 10643 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 169 key (13577817309184 METADATA_ITEM 0) itemoff 10610 itemsize 33
		refs 1 gen 1443709 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 170 key (13577817325568 METADATA_ITEM 0) itemoff 10577 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 171 key (13577817358336 METADATA_ITEM 0) itemoff 10544 itemsize 33
		refs 1 gen 1500489 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 172 key (13577817374720 METADATA_ITEM 0) itemoff 10511 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 173 key (13577817391104 METADATA_ITEM 0) itemoff 10478 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 174 key (13577817407488 METADATA_ITEM 0) itemoff 10445 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 175 key (13577817423872 METADATA_ITEM 0) itemoff 10412 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 176 key (13577817440256 METADATA_ITEM 0) itemoff 10379 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 177 key (13577817456640 METADATA_ITEM 0) itemoff 10346 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 178 key (13577817473024 METADATA_ITEM 0) itemoff 10313 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 179 key (13577817489408 METADATA_ITEM 0) itemoff 10280 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 180 key (13577817505792 METADATA_ITEM 0) itemoff 10247 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 181 key (13577817522176 METADATA_ITEM 0) itemoff 10214 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 182 key (13577817538560 METADATA_ITEM 0) itemoff 10181 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 183 key (13577817554944 METADATA_ITEM 0) itemoff 10148 itemsize 33
		refs 1 gen 1534021 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 184 key (13577817571328 METADATA_ITEM 0) itemoff 10115 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 185 key (13577817587712 METADATA_ITEM 0) itemoff 10082 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 186 key (13577817604096 METADATA_ITEM 0) itemoff 10049 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 187 key (13577817620480 METADATA_ITEM 0) itemoff 10016 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 188 key (13577817636864 METADATA_ITEM 0) itemoff 9983 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 189 key (13577817653248 METADATA_ITEM 0) itemoff 9950 itemsize 33
		refs 1 gen 1406255 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 190 key (13577817669632 METADATA_ITEM 0) itemoff 9917 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 191 key (13577817686016 METADATA_ITEM 0) itemoff 9884 itemsize 33
		refs 1 gen 1509032 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 192 key (13577817702400 METADATA_ITEM 0) itemoff 9851 itemsize 33
		refs 1 gen 1369803 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 193 key (13577817718784 METADATA_ITEM 0) itemoff 9818 itemsize 33
		refs 1 gen 1369803 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 194 key (13577817735168 METADATA_ITEM 0) itemoff 9785 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 195 key (13577817751552 METADATA_ITEM 0) itemoff 9752 itemsize 33
		refs 1 gen 1406255 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 196 key (13577817767936 METADATA_ITEM 0) itemoff 9719 itemsize 33
		refs 1 gen 1534021 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 197 key (13577817784320 METADATA_ITEM 0) itemoff 9686 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 198 key (13577817800704 METADATA_ITEM 0) itemoff 9653 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 199 key (13577817817088 METADATA_ITEM 0) itemoff 9620 itemsize 33
		refs 1 gen 1587580 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 200 key (13577817833472 METADATA_ITEM 0) itemoff 9587 itemsize 33
		refs 1 gen 1390121 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 201 key (13577817849856 METADATA_ITEM 0) itemoff 9554 itemsize 33
		refs 1 gen 1369805 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 202 key (13577817866240 METADATA_ITEM 0) itemoff 9521 itemsize 33
		refs 1 gen 1369805 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 203 key (13577817882624 METADATA_ITEM 0) itemoff 9488 itemsize 33
		refs 1 gen 1369805 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
leaf 364866551808 items 123 free space 55 generation 1453942 owner ROOT_TREE
leaf 364866551808 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
	item 0 key (541 INODE_ITEM 0) itemoff 16123 itemsize 160
		generation 224 transid 224 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746358.640866684 (2018-02-15 17:59:18)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 1 key (541 EXTENT_DATA 0) itemoff 16070 itemsize 53
		generation 224 type 1 (regular)
		extent data disk byte 307043405824 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 2 key (542 INODE_ITEM 0) itemoff 15910 itemsize 160
		generation 224 transid 224 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746358.640866684 (2018-02-15 17:59:18)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 3 key (542 EXTENT_DATA 0) itemoff 15857 itemsize 53
		generation 224 type 1 (regular)
		extent data disk byte 307043667968 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 4 key (543 INODE_ITEM 0) itemoff 15697 itemsize 160
		generation 224 transid 224 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746358.640866684 (2018-02-15 17:59:18)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 5 key (543 EXTENT_DATA 0) itemoff 15644 itemsize 53
		generation 224 type 1 (regular)
		extent data disk byte 307043930112 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 6 key (544 INODE_ITEM 0) itemoff 15484 itemsize 160
		generation 1448952 transid 1448952 size 262144 nbytes 118226944
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 451 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699261.678203037 (2020-07-13 21:01:01)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 7 key (544 EXTENT_DATA 0) itemoff 15431 itemsize 53
		generation 1448952 type 1 (regular)
		extent data disk byte 385910050816 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 8 key (545 INODE_ITEM 0) itemoff 15271 itemsize 160
		generation 228 transid 228 size 262144 nbytes 1310720
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746485.615047014 (2018-02-15 18:01:25)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 9 key (545 EXTENT_DATA 0) itemoff 15218 itemsize 53
		generation 228 type 1 (regular)
		extent data disk byte 311951056896 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 10 key (546 INODE_ITEM 0) itemoff 15058 itemsize 160
		generation 231 transid 231 size 262144 nbytes 1835008
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 7 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746581.861669265 (2018-02-15 18:03:01)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 11 key (546 EXTENT_DATA 0) itemoff 15005 itemsize 53
		generation 231 type 1 (regular)
		extent data disk byte 315042164736 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 12 key (547 INODE_ITEM 0) itemoff 14845 itemsize 160
		generation 228 transid 228 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746485.615047014 (2018-02-15 18:01:25)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 13 key (547 EXTENT_DATA 0) itemoff 14792 itemsize 53
		generation 228 type 1 (regular)
		extent data disk byte 313239584768 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 14 key (548 INODE_ITEM 0) itemoff 14632 itemsize 160
		generation 230 transid 230 size 262144 nbytes 1310720
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746551.226107662 (2018-02-15 18:02:31)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 15 key (548 EXTENT_DATA 0) itemoff 14579 itemsize 53
		generation 230 type 1 (regular)
		extent data disk byte 315171475456 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 16 key (549 INODE_ITEM 0) itemoff 14419 itemsize 160
		generation 229 transid 229 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746520.422548598 (2018-02-15 18:02:00)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 17 key (549 EXTENT_DATA 0) itemoff 14366 itemsize 53
		generation 229 type 1 (regular)
		extent data disk byte 315041640448 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 18 key (550 INODE_ITEM 0) itemoff 14206 itemsize 160
		generation 230 transid 230 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746551.226107662 (2018-02-15 18:02:31)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 19 key (550 EXTENT_DATA 0) itemoff 14153 itemsize 53
		generation 230 type 1 (regular)
		extent data disk byte 315171737600 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 20 key (551 INODE_ITEM 0) itemoff 13993 itemsize 160
		generation 231 transid 231 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746581.861669265 (2018-02-15 18:03:01)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 21 key (551 EXTENT_DATA 0) itemoff 13940 itemsize 53
		generation 231 type 1 (regular)
		extent data disk byte 315042426880 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 22 key (552 INODE_ITEM 0) itemoff 13780 itemsize 160
		generation 1448972 transid 1448972 size 262144 nbytes 11272192
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 43 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594699941.400568279 (2020-07-13 21:12:21)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 23 key (552 EXTENT_DATA 0) itemoff 13727 itemsize 53
		generation 1448972 type 1 (regular)
		extent data disk byte 385932996608 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 24 key (553 INODE_ITEM 0) itemoff 13567 itemsize 160
		generation 232 transid 232 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746614.633200448 (2018-02-15 18:03:34)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 25 key (553 EXTENT_DATA 0) itemoff 13514 itemsize 53
		generation 232 type 1 (regular)
		extent data disk byte 320134049792 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 26 key (554 INODE_ITEM 0) itemoff 13354 itemsize 160
		generation 233 transid 233 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746649.448702556 (2018-02-15 18:04:09)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 27 key (554 EXTENT_DATA 0) itemoff 13301 itemsize 53
		generation 233 type 1 (regular)
		extent data disk byte 318224015360 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 28 key (555 INODE_ITEM 0) itemoff 13141 itemsize 160
		generation 255 transid 255 size 262144 nbytes 3145728
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518747394.918080025 (2018-02-15 18:16:34)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 29 key (555 EXTENT_DATA 0) itemoff 13088 itemsize 53
		generation 255 type 1 (regular)
		extent data disk byte 349531496448 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 30 key (556 INODE_ITEM 0) itemoff 12928 itemsize 160
		generation 234 transid 234 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746684.268204777 (2018-02-15 18:04:44)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 31 key (556 EXTENT_DATA 0) itemoff 12875 itemsize 53
		generation 234 type 1 (regular)
		extent data disk byte 320134991872 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 32 key (557 INODE_ITEM 0) itemoff 12715 itemsize 160
		generation 242 transid 242 size 262144 nbytes 2883584
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746956.648316509 (2018-02-15 18:09:16)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 33 key (557 EXTENT_DATA 0) itemoff 12662 itemsize 53
		generation 242 type 1 (regular)
		extent data disk byte 339572670464 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 34 key (558 INODE_ITEM 0) itemoff 12502 itemsize 160
		generation 235 transid 235 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746719.87707165 (2018-02-15 18:05:19)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 35 key (558 EXTENT_DATA 0) itemoff 12449 itemsize 53
		generation 235 type 1 (regular)
		extent data disk byte 325533712384 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 36 key (559 INODE_ITEM 0) itemoff 12289 itemsize 160
		generation 243 transid 243 size 262144 nbytes 2883584
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746991.467820159 (2018-02-15 18:09:51)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 37 key (559 EXTENT_DATA 0) itemoff 12236 itemsize 53
		generation 243 type 1 (regular)
		extent data disk byte 340941766656 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 38 key (560 INODE_ITEM 0) itemoff 12076 itemsize 160
		generation 235 transid 235 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746719.87707165 (2018-02-15 18:05:19)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 39 key (560 EXTENT_DATA 0) itemoff 12023 itemsize 53
		generation 235 type 1 (regular)
		extent data disk byte 325534236672 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 40 key (561 INODE_ITEM 0) itemoff 11863 itemsize 160
		generation 235 transid 235 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746719.91707108 (2018-02-15 18:05:19)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 41 key (561 EXTENT_DATA 0) itemoff 11810 itemsize 53
		generation 235 type 1 (regular)
		extent data disk byte 325534498816 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 42 key (562 INODE_ITEM 0) itemoff 11650 itemsize 160
		generation 256 transid 256 size 262144 nbytes 2359296
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518747429.853583893 (2018-02-15 18:17:09)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 43 key (562 EXTENT_DATA 0) itemoff 11597 itemsize 53
		generation 256 type 1 (regular)
		extent data disk byte 350605619200 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 44 key (563 INODE_ITEM 0) itemoff 11437 itemsize 160
		generation 243 transid 243 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746991.467820159 (2018-02-15 18:09:51)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 45 key (563 EXTENT_DATA 0) itemoff 11384 itemsize 53
		generation 243 type 1 (regular)
		extent data disk byte 340942028800 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 46 key (564 INODE_ITEM 0) itemoff 11224 itemsize 160
		generation 243 transid 243 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746991.467820159 (2018-02-15 18:09:51)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 47 key (564 EXTENT_DATA 0) itemoff 11171 itemsize 53
		generation 243 type 1 (regular)
		extent data disk byte 341099249664 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 48 key (565 INODE_ITEM 0) itemoff 11011 itemsize 160
		generation 245 transid 245 size 262144 nbytes 2359296
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518747054.954915557 (2018-02-15 18:10:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 49 key (565 EXTENT_DATA 0) itemoff 10958 itemsize 53
		generation 245 type 1 (regular)
		extent data disk byte 344332042240 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 50 key (566 INODE_ITEM 0) itemoff 10798 itemsize 160
		generation 243 transid 243 size 262144 nbytes 1310720
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746991.467820159 (2018-02-15 18:09:51)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 51 key (566 EXTENT_DATA 0) itemoff 10745 itemsize 53
		generation 243 type 1 (regular)
		extent data disk byte 341099773952 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 52 key (567 INODE_ITEM 0) itemoff 10585 itemsize 160
		generation 243 transid 243 size 262144 nbytes 1310720
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746991.471820102 (2018-02-15 18:09:51)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 53 key (567 EXTENT_DATA 0) itemoff 10532 itemsize 53
		generation 243 type 1 (regular)
		extent data disk byte 341100036096 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 54 key (568 INODE_ITEM 0) itemoff 10372 itemsize 160
		generation 243 transid 243 size 262144 nbytes 1310720
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746991.471820102 (2018-02-15 18:09:51)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 55 key (568 EXTENT_DATA 0) itemoff 10319 itemsize 53
		generation 243 type 1 (regular)
		extent data disk byte 341100298240 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 56 key (569 INODE_ITEM 0) itemoff 10159 itemsize 160
		generation 243 transid 243 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746991.471820102 (2018-02-15 18:09:51)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 57 key (569 EXTENT_DATA 0) itemoff 10106 itemsize 53
		generation 243 type 1 (regular)
		extent data disk byte 341100560384 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 58 key (570 INODE_ITEM 0) itemoff 9946 itemsize 160
		generation 246 transid 246 size 262144 nbytes 2097152
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 8 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518747087.718448921 (2018-02-15 18:11:27)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 59 key (570 EXTENT_DATA 0) itemoff 9893 itemsize 53
		generation 246 type 1 (regular)
		extent data disk byte 346001625088 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 60 key (571 INODE_ITEM 0) itemoff 9733 itemsize 160
		generation 243 transid 243 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746991.471820102 (2018-02-15 18:09:51)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 61 key (571 EXTENT_DATA 0) itemoff 9680 itemsize 53
		generation 243 type 1 (regular)
		extent data disk byte 341101084672 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 62 key (572 INODE_ITEM 0) itemoff 9520 itemsize 160
		generation 243 transid 243 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746991.471820102 (2018-02-15 18:09:51)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 63 key (572 EXTENT_DATA 0) itemoff 9467 itemsize 53
		generation 243 type 1 (regular)
		extent data disk byte 341101346816 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 64 key (573 INODE_ITEM 0) itemoff 9307 itemsize 160
		generation 243 transid 243 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746991.471820102 (2018-02-15 18:09:51)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 65 key (573 EXTENT_DATA 0) itemoff 9254 itemsize 53
		generation 243 type 1 (regular)
		extent data disk byte 341101608960 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 66 key (574 INODE_ITEM 0) itemoff 9094 itemsize 160
		generation 243 transid 243 size 262144 nbytes 524288
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518746991.471820102 (2018-02-15 18:09:51)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 67 key (574 EXTENT_DATA 0) itemoff 9041 itemsize 53
		generation 243 type 1 (regular)
		extent data disk byte 341101871104 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 68 key (575 INODE_ITEM 0) itemoff 8881 itemsize 160
		generation 257 transid 257 size 262144 nbytes 3670016
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 14 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518747464.617090343 (2018-02-15 18:17:44)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 69 key (575 EXTENT_DATA 0) itemoff 8828 itemsize 53
		generation 257 type 1 (regular)
		extent data disk byte 350604308480 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 70 key (576 INODE_ITEM 0) itemoff 8668 itemsize 160
		generation 245 transid 245 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518747054.954915557 (2018-02-15 18:10:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 71 key (576 EXTENT_DATA 0) itemoff 8615 itemsize 53
		generation 245 type 1 (regular)
		extent data disk byte 344332566528 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 72 key (577 INODE_ITEM 0) itemoff 8455 itemsize 160
		generation 246 transid 246 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518747087.718448921 (2018-02-15 18:11:27)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 73 key (577 EXTENT_DATA 0) itemoff 8402 itemsize 53
		generation 246 type 1 (regular)
		extent data disk byte 346002149376 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 74 key (578 INODE_ITEM 0) itemoff 8242 itemsize 160
		generation 246 transid 246 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518747087.718448921 (2018-02-15 18:11:27)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 75 key (578 EXTENT_DATA 0) itemoff 8189 itemsize 53
		generation 246 type 1 (regular)
		extent data disk byte 346002411520 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 76 key (579 INODE_ITEM 0) itemoff 8029 itemsize 160
		generation 247 transid 247 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518747120.485982364 (2018-02-15 18:12:00)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 77 key (579 EXTENT_DATA 0) itemoff 7976 itemsize 53
		generation 247 type 1 (regular)
		extent data disk byte 347432411136 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 78 key (580 INODE_ITEM 0) itemoff 7816 itemsize 160
		generation 247 transid 247 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518747120.485982364 (2018-02-15 18:12:00)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 79 key (580 EXTENT_DATA 0) itemoff 7763 itemsize 53
		generation 247 type 1 (regular)
		extent data disk byte 347432673280 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 80 key (581 INODE_ITEM 0) itemoff 7603 itemsize 160
		generation 1448974 transid 1448974 size 262144 nbytes 11272192
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 43 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594700018.221151781 (2020-07-13 21:13:38)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 81 key (581 EXTENT_DATA 0) itemoff 7550 itemsize 53
		generation 1448974 type 1 (regular)
		extent data disk byte 385934254080 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 82 key (582 INODE_ITEM 0) itemoff 7390 itemsize 160
		generation 256 transid 256 size 262144 nbytes 1048576
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518747429.853583893 (2018-02-15 18:17:09)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 83 key (582 EXTENT_DATA 0) itemoff 7337 itemsize 53
		generation 256 type 1 (regular)
		extent data disk byte 352508796928 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 84 key (583 INODE_ITEM 0) itemoff 7177 itemsize 160
		generation 384 transid 384 size 262144 nbytes 3670016
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 14 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518751904.682990094 (2018-02-15 19:31:44)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 85 key (583 EXTENT_DATA 0) itemoff 7124 itemsize 53
		generation 384 type 1 (regular)
		extent data disk byte 430598922240 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 86 key (584 INODE_ITEM 0) itemoff 6964 itemsize 160
		generation 263 transid 263 size 262144 nbytes 4194304
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 16 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518747669.350186333 (2018-02-15 18:21:09)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 87 key (584 EXTENT_DATA 0) itemoff 6911 itemsize 53
		generation 263 type 1 (regular)
		extent data disk byte 353824210944 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 88 key (585 INODE_ITEM 0) itemoff 6751 itemsize 160
		generation 1448979 transid 1448979 size 262144 nbytes 13369344
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 51 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594700214.808521242 (2020-07-13 21:16:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 89 key (585 EXTENT_DATA 0) itemoff 6698 itemsize 53
		generation 1448979 type 1 (regular)
		extent data disk byte 385938366464 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 90 key (586 INODE_ITEM 0) itemoff 6538 itemsize 160
		generation 385 transid 385 size 262144 nbytes 5505024
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 21 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518751939.550509777 (2018-02-15 19:32:19)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 91 key (586 EXTENT_DATA 0) itemoff 6485 itemsize 53
		generation 385 type 1 (regular)
		extent data disk byte 612341153792 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 92 key (587 INODE_ITEM 0) itemoff 6325 itemsize 160
		generation 1448979 transid 1448979 size 262144 nbytes 44302336
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 169 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594700214.808521242 (2020-07-13 21:16:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 93 key (587 EXTENT_DATA 0) itemoff 6272 itemsize 53
		generation 1448979 type 1 (regular)
		extent data disk byte 385938628608 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 94 key (588 INODE_ITEM 0) itemoff 6112 itemsize 160
		generation 355 transid 355 size 262144 nbytes 15728640
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 60 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518750929.764423640 (2018-02-15 19:15:29)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 95 key (588 EXTENT_DATA 0) itemoff 6059 itemsize 53
		generation 355 type 1 (regular)
		extent data disk byte 371543683072 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 96 key (589 INODE_ITEM 0) itemoff 5899 itemsize 160
		generation 285 transid 285 size 262144 nbytes 6029312
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 23 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518748435.303357430 (2018-02-15 18:33:55)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 97 key (589 EXTENT_DATA 0) itemoff 5846 itemsize 53
		generation 285 type 1 (regular)
		extent data disk byte 386576044032 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 98 key (590 INODE_ITEM 0) itemoff 5686 itemsize 160
		generation 385 transid 385 size 262144 nbytes 10747904
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 41 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518751939.554509722 (2018-02-15 19:32:19)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 99 key (590 EXTENT_DATA 0) itemoff 5633 itemsize 53
		generation 385 type 1 (regular)
		extent data disk byte 613844824064 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 100 key (591 INODE_ITEM 0) itemoff 5473 itemsize 160
		generation 353 transid 353 size 262144 nbytes 8388608
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 32 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518750860.313380907 (2018-02-15 19:14:20)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 101 key (591 EXTENT_DATA 0) itemoff 5420 itemsize 53
		generation 353 type 1 (regular)
		extent data disk byte 369396273152 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 102 key (592 INODE_ITEM 0) itemoff 5260 itemsize 160
		generation 285 transid 285 size 262144 nbytes 2359296
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518748435.307357375 (2018-02-15 18:33:55)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 103 key (592 EXTENT_DATA 0) itemoff 5207 itemsize 53
		generation 285 type 1 (regular)
		extent data disk byte 390869721088 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 104 key (593 INODE_ITEM 0) itemoff 5047 itemsize 160
		generation 293 transid 293 size 262144 nbytes 4456448
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 17 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518748722.27313744 (2018-02-15 18:38:42)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 105 key (593 EXTENT_DATA 0) itemoff 4994 itemsize 53
		generation 293 type 1 (regular)
		extent data disk byte 390861172736 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 106 key (594 INODE_ITEM 0) itemoff 4834 itemsize 160
		generation 355 transid 355 size 262144 nbytes 14942208
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 57 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518750929.768423587 (2018-02-15 19:15:29)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 107 key (594 EXTENT_DATA 0) itemoff 4781 itemsize 53
		generation 355 type 1 (regular)
		extent data disk byte 374764580864 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 108 key (595 INODE_ITEM 0) itemoff 4621 itemsize 160
		generation 311 transid 311 size 262144 nbytes 1310720
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518749369.346217917 (2018-02-15 18:49:29)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 109 key (595 EXTENT_DATA 0) itemoff 4568 itemsize 53
		generation 311 type 1 (regular)
		extent data disk byte 424157044736 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 110 key (596 INODE_ITEM 0) itemoff 4408 itemsize 160
		generation 1453942 transid 1453942 size 262144 nbytes 13369344
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 51 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594936992.288440916 (2020-07-16 15:03:12)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 111 key (596 EXTENT_DATA 0) itemoff 4355 itemsize 53
		generation 1453942 type 1 (regular)
		extent data disk byte 749577441280 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 112 key (597 INODE_ITEM 0) itemoff 4195 itemsize 160
		generation 354 transid 354 size 262144 nbytes 15204352
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 58 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518750894.952903454 (2018-02-15 19:14:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 113 key (597 EXTENT_DATA 0) itemoff 4142 itemsize 53
		generation 354 type 1 (regular)
		extent data disk byte 390868410368 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 114 key (598 INODE_ITEM 0) itemoff 3982 itemsize 160
		generation 357 transid 357 size 262144 nbytes 5242880
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 20 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518751001.451435601 (2018-02-15 19:16:41)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 115 key (598 EXTENT_DATA 0) itemoff 3929 itemsize 53
		generation 357 type 1 (regular)
		extent data disk byte 390863429632 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 116 key (599 INODE_ITEM 0) itemoff 3769 itemsize 160
		generation 354 transid 354 size 262144 nbytes 4194304
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 16 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518750894.952903454 (2018-02-15 19:14:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 117 key (599 EXTENT_DATA 0) itemoff 3716 itemsize 53
		generation 354 type 1 (regular)
		extent data disk byte 400534085632 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 118 key (600 INODE_ITEM 0) itemoff 3556 itemsize 160
		generation 311 transid 311 size 262144 nbytes 786432
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518749369.346217917 (2018-02-15 18:49:29)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 119 key (600 EXTENT_DATA 0) itemoff 3503 itemsize 53
		generation 311 type 1 (regular)
		extent data disk byte 426303086592 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 120 key (601 INODE_ITEM 0) itemoff 3343 itemsize 160
		generation 1448976 transid 1448976 size 262144 nbytes 15466496
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 59 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1594700101.145491792 (2020-07-13 21:15:01)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 121 key (601 EXTENT_DATA 0) itemoff 3290 itemsize 53
		generation 1448976 type 1 (regular)
		extent data disk byte 385936875520 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 122 key (602 INODE_ITEM 0) itemoff 3130 itemsize 160
		generation 354 transid 354 size 262144 nbytes 13631488
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 52 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1518750894.956903399 (2018-02-15 19:14:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
parent transid verify failed on 11970666151936 wanted 1601669 found 1602194
checksum verify failed on 11970666151936 wanted 0x2bb9614c found 0xea514a34
checksum verify failed on 11970666151936 wanted 0x2bb9614c found 0xea514a34
bad tree block 11970666151936, bad level, 234 > 8
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  0:58                                                                                       ` Marc MERLIN
@ 2022-04-05  1:06                                                                                         ` Josef Bacik
  2022-04-05  1:16                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05  1:06 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Apr 4, 2022 at 8:58 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Apr 04, 2022 at 08:39:14PM -0400, Josef Bacik wrote:
> > On Mon, Apr 4, 2022 at 8:28 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Mon, Apr 04, 2022 at 08:24:55PM -0400, Josef Bacik wrote:
> > > > > Binary identical after rebuild.
> > > >
> > > > Sigh time for printf sanity checks, thanks,
> > >
> >
> > I'm dumb, try again please, thanks,
>
> progress :)

Ok, lets try

btrfs inspect-internal dump-tree -b 13577779511296

and see what that gives us, the root we think is ok is missing the
part with the actual root items.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  1:06                                                                                         ` Josef Bacik
@ 2022-04-05  1:16                                                                                           ` Marc MERLIN
  2022-04-05  1:22                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05  1:16 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Apr 04, 2022 at 09:06:01PM -0400, Josef Bacik wrote:
> On Mon, Apr 4, 2022 at 8:58 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Mon, Apr 04, 2022 at 08:39:14PM -0400, Josef Bacik wrote:
> > > On Mon, Apr 4, 2022 at 8:28 PM Marc MERLIN <marc@merlins.org> wrote:
> > > >
> > > > On Mon, Apr 04, 2022 at 08:24:55PM -0400, Josef Bacik wrote:
> > > > > > Binary identical after rebuild.
> > > > >
> > > > > Sigh time for printf sanity checks, thanks,
> > > >
> > >
> > > I'm dumb, try again please, thanks,
> >
> > progress :)
> 
> Ok, lets try
> 
> btrfs inspect-internal dump-tree -b 13577779511296
> 
> and see what that gives us, the root we think is ok is missing the
> part with the actual root items.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 13577779511296            
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
FS_INFO IS 0x55de98e14470
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x55de98e14470
btrfs-progs v5.16.2 
leaf 13577779511296 items 201 free space 1149 generation 1602242 owner EXTENT_TREE
leaf 13577779511296 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
	item 0 key (747913936896 EXTENT_ITEM 16384) itemoff 16230 itemsize 53
		refs 1 gen 1453690 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12953489408 count 1
	item 1 key (747913953280 EXTENT_ITEM 4096) itemoff 16177 itemsize 53
		refs 1 gen 1453690 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12953620480 count 1
	item 2 key (747913957376 EXTENT_ITEM 16384) itemoff 16124 itemsize 53
		refs 1 gen 1453690 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12953645056 count 1
	item 3 key (747913973760 EXTENT_ITEM 20480) itemoff 16071 itemsize 53
		refs 1 gen 1453690 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12953776128 count 1
	item 4 key (747913994240 EXTENT_ITEM 12288) itemoff 16018 itemsize 53
		refs 1 gen 1453690 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12953907200 count 1
	item 5 key (747914006528 EXTENT_ITEM 16384) itemoff 15965 itemsize 53
		refs 1 gen 1453690 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12953985024 count 1
	item 6 key (747914022912 EXTENT_ITEM 20480) itemoff 15912 itemsize 53
		refs 1 gen 1453690 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12954083328 count 1
	item 7 key (747914043392 EXTENT_ITEM 20480) itemoff 15859 itemsize 53
		refs 1 gen 1453690 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12954214400 count 1
	item 8 key (747914063872 EXTENT_ITEM 4096) itemoff 15806 itemsize 53
		refs 1 gen 1453690 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12954345472 count 1
	item 9 key (747914067968 EXTENT_ITEM 9043968) itemoff 15753 itemsize 53
		refs 1 gen 1467378 flags DATA
		extent data backref root 11288 objectid 38180 offset 413532160 count 1
	item 10 key (747923111936 EXTENT_ITEM 36864) itemoff 15716 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 11 key (747923148800 EXTENT_ITEM 36864) itemoff 15679 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 12 key (747923210240 EXTENT_ITEM 28672) itemoff 15626 itemsize 53
		refs 1 gen 1539316 flags DATA
		extent data backref root 11288 objectid 72889 offset 169021440 count 1
	item 13 key (747923238912 EXTENT_ITEM 131072) itemoff 15573 itemsize 53
		refs 1 gen 1486211 flags DATA
		extent data backref root 11288 objectid 39190 offset 0 count 1
	item 14 key (747923369984 EXTENT_ITEM 4096) itemoff 15520 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5733089280 count 1
	item 15 key (747923374080 EXTENT_ITEM 262144) itemoff 15467 itemsize 53
		refs 1 gen 1472819 flags DATA
		extent data backref root ROOT_TREE objectid 1377 offset 0 count 1
	item 16 key (747923636224 EXTENT_ITEM 53248) itemoff 15414 itemsize 53
		refs 1 gen 1509130 flags DATA
		extent data backref root 11288 objectid 49146 offset 0 count 1
	item 17 key (747923689472 EXTENT_ITEM 57344) itemoff 15361 itemsize 53
		refs 1 gen 1509130 flags DATA
		extent data backref root 11288 objectid 49147 offset 0 count 1
	item 18 key (747923746816 EXTENT_ITEM 8192) itemoff 15308 itemsize 53
		refs 1 gen 1509156 flags DATA
		extent data backref root 11288 objectid 50588 offset 0 count 1
	item 19 key (747923755008 EXTENT_ITEM 8192) itemoff 15255 itemsize 53
		refs 1 gen 1509156 flags DATA
		extent data backref root 11288 objectid 50613 offset 0 count 1
	item 20 key (747923763200 EXTENT_ITEM 4096) itemoff 15202 itemsize 53
		refs 1 gen 1509865 flags DATA
		extent data backref root 11288 objectid 70849 offset 0 count 1
	item 21 key (747923767296 EXTENT_ITEM 77824) itemoff 15149 itemsize 53
		refs 1 gen 1509129 flags DATA
		extent data backref root 11288 objectid 49044 offset 0 count 1
	item 22 key (747923845120 EXTENT_ITEM 53248) itemoff 15096 itemsize 53
		refs 1 gen 1509130 flags DATA
		extent data backref root 11288 objectid 49148 offset 0 count 1
	item 23 key (747923898368 EXTENT_ITEM 114688) itemoff 15043 itemsize 53
		refs 1 gen 1484194 flags DATA
		extent data backref root 11288 objectid 39120 offset 2883584 count 1
	item 24 key (747924013056 EXTENT_ITEM 36864) itemoff 15006 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 25 key (747924049920 EXTENT_ITEM 36864) itemoff 14969 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 26 key (747924086784 EXTENT_ITEM 36864) itemoff 14932 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 27 key (747924144128 EXTENT_ITEM 8192) itemoff 14879 itemsize 53
		refs 1 gen 1590235 flags DATA
		extent data backref root 11223 objectid 1823997 offset 0 count 1
	item 28 key (747924152320 EXTENT_ITEM 8192) itemoff 14826 itemsize 53
		refs 1 gen 1590235 flags DATA
		extent data backref root 11223 objectid 1823998 offset 0 count 1
	item 29 key (747924160512 EXTENT_ITEM 86016) itemoff 14773 itemsize 53
		refs 1 gen 1508916 flags DATA
		extent data backref root 11288 objectid 40988 offset 393216 count 1
	item 30 key (747924246528 EXTENT_ITEM 114688) itemoff 14720 itemsize 53
		refs 1 gen 1508916 flags DATA
		extent data backref root 11288 objectid 40988 offset 524288 count 1
	item 31 key (747924361216 EXTENT_ITEM 32768) itemoff 14667 itemsize 53
		refs 1 gen 1508973 flags DATA
		extent data backref root 11288 objectid 41258 offset 0 count 1
	item 32 key (747924393984 EXTENT_ITEM 28672) itemoff 14614 itemsize 53
		refs 1 gen 1508973 flags DATA
		extent data backref root 11288 objectid 41261 offset 0 count 1
	item 33 key (747924422656 EXTENT_ITEM 16384) itemoff 14561 itemsize 53
		refs 1 gen 1453686 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12942655488 count 1
	item 34 key (747924439040 EXTENT_ITEM 16384) itemoff 14508 itemsize 53
		refs 1 gen 1453686 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12942786560 count 1
	item 35 key (747924455424 EXTENT_ITEM 196608) itemoff 14455 itemsize 53
		refs 1 gen 1482902 flags DATA
		extent data backref root 11288 objectid 39004 offset 2231795712 count 1
	item 36 key (747924652032 EXTENT_ITEM 53248) itemoff 14402 itemsize 53
		refs 1 gen 1482903 flags DATA
		extent data backref root 11288 objectid 39005 offset 131072 count 1
	item 37 key (747924705280 EXTENT_ITEM 36864) itemoff 14365 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 38 key (747924742144 EXTENT_ITEM 36864) itemoff 14328 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 39 key (747924779008 EXTENT_ITEM 36864) itemoff 14291 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 40 key (747924815872 EXTENT_ITEM 36864) itemoff 14254 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 41 key (747924852736 EXTENT_ITEM 36864) itemoff 14217 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 42 key (747924889600 EXTENT_ITEM 36864) itemoff 14180 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 43 key (747924934656 EXTENT_ITEM 28672) itemoff 14127 itemsize 53
		refs 1 gen 1565039 flags DATA
		extent data backref root 11288 objectid 75169 offset 6160384 count 1
	item 44 key (747924963328 EXTENT_ITEM 4096) itemoff 14074 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5733220352 count 1
	item 45 key (747924967424 EXTENT_ITEM 4096) itemoff 14021 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5733351424 count 1
	item 46 key (747924971520 EXTENT_ITEM 4096) itemoff 13968 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5733482496 count 1
	item 47 key (747924975616 EXTENT_ITEM 4096) itemoff 13915 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5733613568 count 1
	item 48 key (747924979712 EXTENT_ITEM 36864) itemoff 13862 itemsize 53
		refs 1 gen 1597228 flags DATA
		extent data backref root 11221 objectid 414410 offset 0 count 1
	item 49 key (747925016576 EXTENT_ITEM 36864) itemoff 13809 itemsize 53
		refs 1 gen 1597228 flags DATA
		extent data backref root 11221 objectid 414410 offset 131072 count 1
	item 50 key (747925053440 EXTENT_ITEM 36864) itemoff 13756 itemsize 53
		refs 1 gen 1597228 flags DATA
		extent data backref root 11221 objectid 414410 offset 262144 count 1
	item 51 key (747925090304 EXTENT_ITEM 36864) itemoff 13703 itemsize 53
		refs 1 gen 1597228 flags DATA
		extent data backref root 11221 objectid 414410 offset 393216 count 1
	item 52 key (747925127168 EXTENT_ITEM 36864) itemoff 13650 itemsize 53
		refs 1 gen 1597228 flags DATA
		extent data backref root 11221 objectid 414410 offset 524288 count 1
	item 53 key (747925164032 EXTENT_ITEM 36864) itemoff 13597 itemsize 53
		refs 1 gen 1597228 flags DATA
		extent data backref root 11221 objectid 414410 offset 655360 count 1
	item 54 key (747925200896 EXTENT_ITEM 36864) itemoff 13544 itemsize 53
		refs 1 gen 1597228 flags DATA
		extent data backref root 11221 objectid 414410 offset 786432 count 1
	item 55 key (747925241856 EXTENT_ITEM 155648) itemoff 13491 itemsize 53
		refs 1 gen 1509071 flags DATA
		extent data backref root 11288 objectid 46649 offset 0 count 1
	item 56 key (747925397504 EXTENT_ITEM 86016) itemoff 13454 itemsize 37
		refs 1 gen 1509093 flags DATA
		shared data backref parent 12512079577088 count 1
	item 57 key (747925483520 EXTENT_ITEM 20480) itemoff 13401 itemsize 53
		refs 1 gen 1509156 flags DATA
		extent data backref root 11288 objectid 50590 offset 0 count 1
	item 58 key (747925504000 EXTENT_ITEM 36864) itemoff 13364 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 59 key (747925540864 EXTENT_ITEM 36864) itemoff 13327 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 60 key (747925577728 EXTENT_ITEM 36864) itemoff 13290 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 61 key (747925614592 EXTENT_ITEM 36864) itemoff 13253 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 62 key (747925651456 EXTENT_ITEM 36864) itemoff 13216 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 63 key (747925700608 EXTENT_ITEM 4096) itemoff 13163 itemsize 53
		refs 1 gen 1590235 flags DATA
		extent data backref root 11223 objectid 1824000 offset 0 count 1
	item 64 key (747925704704 EXTENT_ITEM 4096) itemoff 13110 itemsize 53
		refs 1 gen 1590235 flags DATA
		extent data backref root 11223 objectid 1824005 offset 0 count 1
	item 65 key (747925708800 EXTENT_ITEM 4096) itemoff 13057 itemsize 53
		refs 1 gen 1590235 flags DATA
		extent data backref root 11223 objectid 1824007 offset 0 count 1
	item 66 key (747925712896 EXTENT_ITEM 12288) itemoff 13004 itemsize 53
		refs 1 gen 1553656 flags DATA
		extent data backref root 11288 objectid 74580 offset 0 count 1
	item 67 key (747925725184 EXTENT_ITEM 4096) itemoff 12951 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5733744640 count 1
	item 68 key (747925729280 EXTENT_ITEM 49152) itemoff 12898 itemsize 53
		refs 1 gen 1484713 flags DATA
		extent data backref root 11288 objectid 39130 offset 0 count 1
	item 69 key (747925778432 EXTENT_ITEM 57344) itemoff 12845 itemsize 53
		refs 1 gen 1484739 flags DATA
		extent data backref root 11288 objectid 39131 offset 727973888 count 1
	item 70 key (747925835776 EXTENT_ITEM 32768) itemoff 12808 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510516736 count 1
	item 71 key (747925868544 EXTENT_ITEM 4096) itemoff 12755 itemsize 53
		refs 1 gen 1590235 flags DATA
		extent data backref root 11223 objectid 1824014 offset 0 count 1
	item 72 key (747925872640 EXTENT_ITEM 4096) itemoff 12702 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5733875712 count 1
	item 73 key (747925876736 EXTENT_ITEM 4096) itemoff 12649 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5734006784 count 1
	item 74 key (747925880832 EXTENT_ITEM 4096) itemoff 12596 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5734137856 count 1
	item 75 key (747925884928 EXTENT_ITEM 4096) itemoff 12543 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5734268928 count 1
	item 76 key (747925889024 EXTENT_ITEM 4096) itemoff 12490 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5734400000 count 1
	item 77 key (747925893120 EXTENT_ITEM 4096) itemoff 12437 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5734531072 count 1
	item 78 key (747925897216 EXTENT_ITEM 118784) itemoff 12384 itemsize 53
		refs 1 gen 1484194 flags DATA
		extent data backref root 11288 objectid 39120 offset 3014656 count 1
	item 79 key (747926016000 EXTENT_ITEM 114688) itemoff 12331 itemsize 53
		refs 1 gen 1484194 flags DATA
		extent data backref root 11288 objectid 39120 offset 3145728 count 1
	item 80 key (747926130688 EXTENT_ITEM 36864) itemoff 12294 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 81 key (747926167552 EXTENT_ITEM 36864) itemoff 12257 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 82 key (747926204416 EXTENT_ITEM 36864) itemoff 12220 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 83 key (747926241280 EXTENT_ITEM 36864) itemoff 12183 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 84 key (747926278144 EXTENT_ITEM 36864) itemoff 12146 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 85 key (747926315008 EXTENT_ITEM 36864) itemoff 12109 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 86 key (747926360064 EXTENT_ITEM 20480) itemoff 12056 itemsize 53
		refs 1 gen 1590234 flags DATA
		extent data backref root 11223 objectid 1823985 offset 0 count 1
	item 87 key (747926380544 EXTENT_ITEM 4096) itemoff 12003 itemsize 53
		refs 1 gen 1590235 flags DATA
		extent data backref root 11223 objectid 1824015 offset 0 count 1
	item 88 key (747926384640 EXTENT_ITEM 4096) itemoff 11950 itemsize 53
		refs 1 gen 1590235 flags DATA
		extent data backref root 11223 objectid 1824020 offset 0 count 1
	item 89 key (747926388736 EXTENT_ITEM 4096) itemoff 11897 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5734662144 count 1
	item 90 key (747926392832 EXTENT_ITEM 36864) itemoff 11860 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 91 key (747926458368 EXTENT_ITEM 4096) itemoff 11807 itemsize 53
		refs 1 gen 1590235 flags DATA
		extent data backref root 11223 objectid 1824030 offset 0 count 1
	item 92 key (747926462464 EXTENT_ITEM 4096) itemoff 11754 itemsize 53
		refs 1 gen 1590235 flags DATA
		extent data backref root 11223 objectid 1824032 offset 0 count 1
	item 93 key (747926466560 EXTENT_ITEM 4096) itemoff 11701 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5734793216 count 1
	item 94 key (747926470656 EXTENT_ITEM 4096) itemoff 11648 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5734924288 count 1
	item 95 key (747926474752 EXTENT_ITEM 53248) itemoff 11595 itemsize 53
		refs 1 gen 1483072 flags DATA
		extent data backref root 11288 objectid 39029 offset 0 count 1
	item 96 key (747926528000 EXTENT_ITEM 4096) itemoff 11542 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5735055360 count 1
	item 97 key (747926532096 EXTENT_ITEM 4096) itemoff 11489 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5735186432 count 1
	item 98 key (747926536192 EXTENT_ITEM 4096) itemoff 11436 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5735317504 count 1
	item 99 key (747926540288 EXTENT_ITEM 4096) itemoff 11383 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5735448576 count 1
	item 100 key (747926544384 EXTENT_ITEM 4096) itemoff 11330 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5735579648 count 1
	item 101 key (747926548480 EXTENT_ITEM 4096) itemoff 11277 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5735710720 count 1
	item 102 key (747926552576 EXTENT_ITEM 53248) itemoff 11224 itemsize 53
		refs 1 gen 1482894 flags DATA
		extent data backref root 11288 objectid 39000 offset 1179648 count 1
	item 103 key (747926605824 EXTENT_ITEM 57344) itemoff 11171 itemsize 53
		refs 1 gen 1482894 flags DATA
		extent data backref root 11288 objectid 39000 offset 1310720 count 1
	item 104 key (747926663168 EXTENT_ITEM 53248) itemoff 11118 itemsize 53
		refs 1 gen 1482894 flags DATA
		extent data backref root 11288 objectid 39000 offset 1441792 count 1
	item 105 key (747926716416 EXTENT_ITEM 53248) itemoff 11065 itemsize 53
		refs 1 gen 1482894 flags DATA
		extent data backref root 11288 objectid 39000 offset 1572864 count 1
	item 106 key (747926769664 EXTENT_ITEM 32768) itemoff 11012 itemsize 53
		refs 1 gen 1482894 flags DATA
		extent data backref root 11288 objectid 39001 offset 524288 count 1
	item 107 key (747926802432 EXTENT_ITEM 4096) itemoff 10959 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5735841792 count 1
	item 108 key (747926806528 EXTENT_ITEM 4096) itemoff 10906 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5735972864 count 1
	item 109 key (747926810624 EXTENT_ITEM 4096) itemoff 10853 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5736103936 count 1
	item 110 key (747926814720 EXTENT_ITEM 262144) itemoff 10800 itemsize 53
		refs 1 gen 1474024 flags DATA
		extent data backref root ROOT_TREE objectid 2566 offset 0 count 1
	item 111 key (747927076864 EXTENT_ITEM 16384) itemoff 10747 itemsize 53
		refs 1 gen 1453686 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12943011840 count 1
	item 112 key (747927093248 EXTENT_ITEM 16384) itemoff 10694 itemsize 53
		refs 1 gen 1453686 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12943142912 count 1
	item 113 key (747927109632 EXTENT_ITEM 8192) itemoff 10641 itemsize 53
		refs 1 gen 1453686 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12943273984 count 1
	item 114 key (747927117824 EXTENT_ITEM 32768) itemoff 10588 itemsize 53
		refs 1 gen 1602242 flags DATA
		extent data backref root 11221 objectid 414456 offset 0 count 1
	item 115 key (747927150592 EXTENT_ITEM 36864) itemoff 10535 itemsize 53
		refs 1 gen 1602242 flags DATA
		extent data backref root 11221 objectid 414456 offset 131072 count 1
	item 116 key (747927187456 EXTENT_ITEM 32768) itemoff 10482 itemsize 53
		refs 1 gen 1602242 flags DATA
		extent data backref root 11221 objectid 414456 offset 262144 count 1
	item 117 key (747927379968 EXTENT_ITEM 229376) itemoff 10429 itemsize 53
		refs 1 gen 1482585 flags DATA
		extent data backref root 11288 objectid 38979 offset 5445386240 count 1
	item 118 key (747927609344 EXTENT_ITEM 32768) itemoff 10376 itemsize 53
		refs 1 gen 1482590 flags DATA
		extent data backref root 11288 objectid 38979 offset 9466732544 count 1
	item 119 key (747927642112 EXTENT_ITEM 229376) itemoff 10323 itemsize 53
		refs 1 gen 1482593 flags DATA
		extent data backref root 11288 objectid 38979 offset 11495415808 count 1
	item 120 key (747927871488 EXTENT_ITEM 32768) itemoff 10286 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510516736 count 1
	item 121 key (747927904256 EXTENT_ITEM 524288) itemoff 10233 itemsize 53
		refs 1 gen 1483575 flags DATA
		extent data backref root 11288 objectid 39051 offset 524288 count 1
	item 122 key (747928428544 EXTENT_ITEM 36864) itemoff 10196 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 123 key (747928494080 EXTENT_ITEM 4096) itemoff 10143 itemsize 53
		refs 1 gen 1590235 flags DATA
		extent data backref root 11223 objectid 1824033 offset 0 count 1
	item 124 key (747928498176 EXTENT_ITEM 4096) itemoff 10090 itemsize 53
		refs 1 gen 1590235 flags DATA
		extent data backref root 11223 objectid 1824036 offset 0 count 1
	item 125 key (747928502272 EXTENT_ITEM 4096) itemoff 10037 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5736235008 count 1
	item 126 key (747928506368 EXTENT_ITEM 4096) itemoff 9984 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5736366080 count 1
	item 127 key (747928510464 EXTENT_ITEM 8192) itemoff 9931 itemsize 53
		refs 1 gen 1553656 flags DATA
		extent data backref root 11288 objectid 74580 offset 393216 count 1
	item 128 key (747928518656 EXTENT_ITEM 8192) itemoff 9878 itemsize 53
		refs 1 gen 1553656 flags DATA
		extent data backref root 11288 objectid 74581 offset 131072 count 1
	item 129 key (747928526848 EXTENT_ITEM 110592) itemoff 9825 itemsize 53
		refs 1 gen 1477712 flags DATA
		extent data backref root 11288 objectid 38340 offset 3955474432 count 1
	item 130 key (747928637440 EXTENT_ITEM 32768) itemoff 9788 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510516736 count 1
	item 131 key (747928670208 EXTENT_ITEM 4096) itemoff 9735 itemsize 53
		refs 1 gen 1590235 flags DATA
		extent data backref root 11223 objectid 1824038 offset 0 count 1
	item 132 key (747928674304 EXTENT_ITEM 8192) itemoff 9682 itemsize 53
		refs 1 gen 1480427 flags DATA
		extent data backref root 11288 objectid 38887 offset 0 count 1
	item 133 key (747928682496 EXTENT_ITEM 8192) itemoff 9629 itemsize 53
		refs 1 gen 1477753 flags DATA
		extent data backref root 11288 objectid 38401 offset 2228224 count 1
	item 134 key (747928690688 EXTENT_ITEM 229376) itemoff 9576 itemsize 53
		refs 1 gen 1482583 flags DATA
		extent data backref root 11288 objectid 38979 offset 5248192512 count 1
	item 135 key (747928920064 EXTENT_ITEM 32768) itemoff 9523 itemsize 53
		refs 1 gen 1482590 flags DATA
		extent data backref root 11288 objectid 38979 offset 9702588416 count 1
	item 136 key (747928952832 EXTENT_ITEM 262144) itemoff 9470 itemsize 53
		refs 1 gen 1482580 flags DATA
		extent data backref root ROOT_TREE objectid 5552 offset 0 count 1
	item 137 key (747929214976 EXTENT_ITEM 16384) itemoff 9417 itemsize 53
		refs 1 gen 1453686 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12943327232 count 1
	item 138 key (747929231360 EXTENT_ITEM 16384) itemoff 9364 itemsize 53
		refs 1 gen 1453686 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12943458304 count 1
	item 139 key (747929247744 EXTENT_ITEM 4096) itemoff 9311 itemsize 53
		refs 1 gen 1453686 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12943589376 count 1
	item 140 key (747929251840 EXTENT_ITEM 524288) itemoff 9258 itemsize 53
		refs 1 gen 1480794 flags DATA
		extent data backref root 11288 objectid 38927 offset 1572864 count 1
	item 141 key (747929776128 EXTENT_ITEM 196608) itemoff 9205 itemsize 53
		refs 1 gen 1482583 flags DATA
		extent data backref root 11288 objectid 38979 offset 5799989248 count 1
	item 142 key (747929972736 EXTENT_ITEM 65536) itemoff 9152 itemsize 53
		refs 1 gen 1482585 flags DATA
		extent data backref root 11288 objectid 38979 offset 6808727552 count 1
	item 143 key (747930038272 EXTENT_ITEM 65536) itemoff 9099 itemsize 53
		refs 1 gen 1482586 flags DATA
		extent data backref root 11288 objectid 38979 offset 6901035008 count 1
	item 144 key (747930103808 EXTENT_ITEM 65536) itemoff 9046 itemsize 53
		refs 1 gen 1482586 flags DATA
		extent data backref root 11288 objectid 38979 offset 7791390720 count 1
	item 145 key (747930169344 EXTENT_ITEM 61440) itemoff 8993 itemsize 53
		refs 1 gen 1530100 flags DATA
		extent data backref root 11288 objectid 72155 offset 367656960 count 1
	item 146 key (747930230784 EXTENT_ITEM 61440) itemoff 8940 itemsize 53
		refs 1 gen 1530100 flags DATA
		extent data backref root 11288 objectid 72155 offset 367788032 count 1
	item 147 key (747930292224 EXTENT_ITEM 8192) itemoff 8887 itemsize 53
		refs 1 gen 1553656 flags DATA
		extent data backref root 11288 objectid 74581 offset 262144 count 1
	item 148 key (747930300416 EXTENT_ITEM 69632) itemoff 8834 itemsize 53
		refs 1 gen 1530011 flags DATA
		extent data backref root 11288 objectid 72146 offset 0 count 1
	item 149 key (747930370048 EXTENT_ITEM 61440) itemoff 8781 itemsize 53
		refs 1 gen 1530025 flags DATA
		extent data backref root 11288 objectid 72148 offset 367656960 count 1
	item 150 key (747930431488 EXTENT_ITEM 131072) itemoff 8728 itemsize 53
		refs 1 gen 1482583 flags DATA
		extent data backref root 11288 objectid 38979 offset 6275604480 count 1
	item 151 key (747930562560 EXTENT_ITEM 229376) itemoff 8675 itemsize 53
		refs 1 gen 1482588 flags DATA
		extent data backref root 11288 objectid 38979 offset 8650186752 count 1
	item 152 key (747930791936 EXTENT_ITEM 32768) itemoff 8622 itemsize 53
		refs 1 gen 1482590 flags DATA
		extent data backref root 11288 objectid 38979 offset 9860608000 count 1
	item 153 key (747930824704 EXTENT_ITEM 196608) itemoff 8569 itemsize 53
		refs 1 gen 1482582 flags DATA
		extent data backref root 11288 objectid 38979 offset 6011453440 count 1
	item 154 key (747931021312 EXTENT_ITEM 65536) itemoff 8516 itemsize 53
		refs 1 gen 1482585 flags DATA
		extent data backref root 11288 objectid 38979 offset 7213383680 count 1
	item 155 key (747931086848 EXTENT_ITEM 94208) itemoff 8463 itemsize 53
		refs 1 gen 1477873 flags DATA
		extent data backref root 11288 objectid 38444 offset 262144 count 1
	item 156 key (747931181056 EXTENT_ITEM 20480) itemoff 8410 itemsize 53
		refs 1 gen 1477873 flags DATA
		extent data backref root 11288 objectid 38444 offset 2621440 count 1
	item 157 key (747931201536 EXTENT_ITEM 114688) itemoff 8357 itemsize 53
		refs 1 gen 1477874 flags DATA
		extent data backref root 11288 objectid 38444 offset 113442816 count 1
	item 158 key (747931316224 EXTENT_ITEM 16384) itemoff 8304 itemsize 53
		refs 1 gen 1477879 flags DATA
		extent data backref root 11288 objectid 38444 offset 848715776 count 1
	item 159 key (747931332608 EXTENT_ITEM 4096) itemoff 8251 itemsize 53
		refs 1 gen 1477880 flags DATA
		extent data backref root 11288 objectid 38444 offset 848728064 count 1
	item 160 key (747931336704 EXTENT_ITEM 12288) itemoff 8198 itemsize 53
		refs 1 gen 1477883 flags DATA
		extent data backref root 11288 objectid 38448 offset 2752512 count 1
	item 161 key (747931348992 EXTENT_ITEM 8192) itemoff 8145 itemsize 53
		refs 1 gen 1453686 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12943601664 count 1
	item 162 key (747931357184 EXTENT_ITEM 229376) itemoff 8092 itemsize 53
		refs 1 gen 1482585 flags DATA
		extent data backref root 11288 objectid 38979 offset 6740770816 count 1
	item 163 key (747931586560 EXTENT_ITEM 32768) itemoff 8039 itemsize 53
		refs 1 gen 1482590 flags DATA
		extent data backref root 11288 objectid 38979 offset 9932337152 count 1
	item 164 key (747931619328 EXTENT_ITEM 36864) itemoff 8002 itemsize 37
		refs 1 gen 1592383 flags DATA
		shared data backref parent 15645802905600 count 1
	item 165 key (747931656192 EXTENT_ITEM 36864) itemoff 7965 itemsize 37
		refs 1 gen 1592383 flags DATA
		shared data backref parent 15645802905600 count 1
	item 166 key (747931693056 EXTENT_ITEM 36864) itemoff 7928 itemsize 37
		refs 1 gen 1592383 flags DATA
		shared data backref parent 15645802905600 count 1
	item 167 key (747931750400 EXTENT_ITEM 36864) itemoff 7891 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 168 key (747931787264 EXTENT_ITEM 36864) itemoff 7854 itemsize 37
		refs 1 gen 1592254 flags DATA
		shared data backref parent 15645510156288 count 1
	item 169 key (747931848704 EXTENT_ITEM 4096) itemoff 7801 itemsize 53
		refs 1 gen 1590235 flags DATA
		extent data backref root 11223 objectid 1824046 offset 0 count 1
	item 170 key (747931852800 EXTENT_ITEM 24576) itemoff 7748 itemsize 53
		refs 1 gen 1539342 flags DATA
		extent data backref root 11288 objectid 72973 offset 327155712 count 1
	item 171 key (747931877376 EXTENT_ITEM 4096) itemoff 7695 itemsize 53
		refs 1 gen 1502140 flags DATA
		extent data backref root 11288 objectid 40707 offset 5736497152 count 1
	item 172 key (747931881472 EXTENT_ITEM 8192) itemoff 7642 itemsize 53
		refs 1 gen 1453686 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12943663104 count 1
	item 173 key (747931889664 EXTENT_ITEM 65536) itemoff 7589 itemsize 53
		refs 1 gen 1482895 flags DATA
		extent data backref root 11288 objectid 39001 offset 370343936 count 1
	item 174 key (747931955200 EXTENT_ITEM 65536) itemoff 7536 itemsize 53
		refs 1 gen 1482895 flags DATA
		extent data backref root 11288 objectid 39001 offset 853999616 count 1
	item 175 key (747932020736 EXTENT_ITEM 98304) itemoff 7483 itemsize 53
		refs 1 gen 1482897 flags DATA
		extent data backref root 11288 objectid 39001 offset 1256620032 count 1
	item 176 key (747932119040 EXTENT_ITEM 32768) itemoff 7430 itemsize 53
		refs 1 gen 1482897 flags DATA
		extent data backref root 11288 objectid 39001 offset 1269202944 count 1
	item 177 key (747932151808 EXTENT_ITEM 139264) itemoff 7377 itemsize 53
		refs 1 gen 1509071 flags DATA
		extent data backref root 11288 objectid 46650 offset 0 count 1
	item 178 key (747932291072 EXTENT_ITEM 122880) itemoff 7340 itemsize 37
		refs 1 gen 1509093 flags DATA
		shared data backref parent 12512079577088 count 1
	item 179 key (747932413952 EXTENT_ITEM 12288) itemoff 7287 itemsize 53
		refs 1 gen 1453686 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12943728640 count 1
	item 180 key (747932426240 EXTENT_ITEM 262144) itemoff 7234 itemsize 53
		refs 1 gen 1472827 flags DATA
		extent data backref root ROOT_TREE objectid 1376 offset 0 count 1
	item 181 key (747932688384 EXTENT_ITEM 262144) itemoff 7181 itemsize 53
		refs 1 gen 1482897 flags DATA
		extent data backref root 11288 objectid 39001 offset 1145307136 count 1
	item 182 key (747932950528 EXTENT_ITEM 40960) itemoff 7128 itemsize 53
		refs 1 gen 1509030 flags DATA
		extent data backref root 11288 objectid 43699 offset 0 count 1
	item 183 key (747932991488 EXTENT_ITEM 53248) itemoff 7075 itemsize 53
		refs 1 gen 1509030 flags DATA
		extent data backref root 11288 objectid 43700 offset 0 count 1
	item 184 key (747933044736 EXTENT_ITEM 69632) itemoff 7022 itemsize 53
		refs 1 gen 1509030 flags DATA
		extent data backref root 11288 objectid 43701 offset 0 count 1
	item 185 key (747933114368 EXTENT_ITEM 77824) itemoff 6969 itemsize 53
		refs 1 gen 1509030 flags DATA
		extent data backref root 11288 objectid 43702 offset 0 count 1
	item 186 key (747933192192 EXTENT_ITEM 262144) itemoff 6916 itemsize 53
		refs 1 gen 1550304 flags DATA
		extent data backref root ROOT_TREE objectid 57451 offset 0 count 1
	item 187 key (747933454336 EXTENT_ITEM 20480) itemoff 6863 itemsize 53
		refs 1 gen 1509156 flags DATA
		extent data backref root 11288 objectid 50591 offset 0 count 1
	item 188 key (747933474816 EXTENT_ITEM 262144) itemoff 6810 itemsize 53
		refs 1 gen 1550313 flags DATA
		extent data backref root ROOT_TREE objectid 57448 offset 0 count 1
	item 189 key (747933736960 EXTENT_ITEM 262144) itemoff 6757 itemsize 53
		refs 1 gen 1472802 flags DATA
		extent data backref root ROOT_TREE objectid 1368 offset 0 count 1
	item 190 key (747933999104 EXTENT_ITEM 16384) itemoff 6704 itemsize 53
		refs 1 gen 1453688 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12949532672 count 1
	item 191 key (747934015488 EXTENT_ITEM 16384) itemoff 6651 itemsize 53
		refs 1 gen 1453688 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12949663744 count 1
	item 192 key (747934031872 EXTENT_ITEM 8192) itemoff 6598 itemsize 53
		refs 1 gen 1453688 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12949794816 count 1
	item 193 key (747934040064 EXTENT_ITEM 8192) itemoff 6545 itemsize 53
		refs 1 gen 1453688 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12949843968 count 1
	item 194 key (747934048256 EXTENT_ITEM 16384) itemoff 6492 itemsize 53
		refs 1 gen 1453688 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12949905408 count 1
	item 195 key (747934064640 EXTENT_ITEM 8192) itemoff 6439 itemsize 53
		refs 1 gen 1453688 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12950036480 count 1
	item 196 key (747934072832 EXTENT_ITEM 12288) itemoff 6386 itemsize 53
		refs 1 gen 1453688 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12950093824 count 1
	item 197 key (747934085120 EXTENT_ITEM 4096) itemoff 6333 itemsize 53
		refs 1 gen 1453688 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12950089728 count 1
	item 198 key (747934089216 EXTENT_ITEM 20480) itemoff 6280 itemsize 53
		refs 1 gen 1453690 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12954370048 count 1
	item 199 key (747934109696 EXTENT_ITEM 20480) itemoff 6227 itemsize 53
		refs 1 gen 1453690 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12954501120 count 1
	item 200 key (747934130176 EXTENT_ITEM 4096) itemoff 6174 itemsize 53
		refs 1 gen 1453690 flags DATA
		extent data backref root FS_TREE objectid 1003 offset 12954632192 count 1

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  1:16                                                                                           ` Marc MERLIN
@ 2022-04-05  1:22                                                                                             ` Josef Bacik
  2022-04-05  1:42                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05  1:22 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Apr 4, 2022 at 9:16 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Apr 04, 2022 at 09:06:01PM -0400, Josef Bacik wrote:
> > On Mon, Apr 4, 2022 at 8:58 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Mon, Apr 04, 2022 at 08:39:14PM -0400, Josef Bacik wrote:
> > > > On Mon, Apr 4, 2022 at 8:28 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > >
> > > > > On Mon, Apr 04, 2022 at 08:24:55PM -0400, Josef Bacik wrote:
> > > > > > > Binary identical after rebuild.
> > > > > >
> > > > > > Sigh time for printf sanity checks, thanks,
> > > > >
> > > >
> > > > I'm dumb, try again please, thanks,
> > >
> > > progress :)
> >
> > Ok, lets try
> >
> > btrfs inspect-internal dump-tree -b 13577779511296
> >
> > and see what that gives us, the root we think is ok is missing the
> > part with the actual root items.  Thanks,
>

Ok lets try again with -b 13577821667328, and if the owner doesn't say
ROOT_TREE try 13577775284224 and then 13577814573056.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  1:22                                                                                             ` Josef Bacik
@ 2022-04-05  1:42                                                                                               ` Marc MERLIN
  2022-04-05  1:55                                                                                                 ` Josef Bacik
                                                                                                                   ` (2 more replies)
  0 siblings, 3 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05  1:42 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Apr 04, 2022 at 09:22:26PM -0400, Josef Bacik wrote:
> Ok lets try again with -b 13577821667328, and if the owner doesn't say
> ROOT_TREE try 13577775284224 and then 13577814573056.  Thanks,

EXTENT_TREE, FS_TREE, and FS_TREE

And shit, I got distracted and sent the text output to
/dev/mapper/dshelf1a, so I clobbered about 30K of the device. 
I'm assuming there was probably something there?

Script started on Mon 04 Apr 2022 18:36:51 PDT
1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m ./btrfs-inspect-internal dump-tree -b 13577821667328 &>/dev/mapper/dshelf1a
btrfs inspect-internal dump-tree: not enough arguments: 0 but at least 1 expected
1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m K./btrfs inspect-internal dump-tree -b 13577821667328 /dev/mapper/dshelf1a
btrfs-progs v5.16.2
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
FS_INFO IS 0x55fe04c74870
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x55fe04c74870
leaf 13577821667328 items 197 free space 4857 generation 1602242 owner EXTENT_TREE
leaf 13577821667328 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
	item 0 key (8733701406720 METADATA_ITEM 0) itemoff 16250 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 1 key (8733701423104 METADATA_ITEM 0) itemoff 16217 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 2 key (8733701439488 METADATA_ITEM 0) itemoff 16184 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 3 key (8733701455872 METADATA_ITEM 0) itemoff 16151 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 4 key (8733701472256 METADATA_ITEM 0) itemoff 16118 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 5 key (8733701488640 METADATA_ITEM 0) itemoff 16085 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 6 key (8733701505024 METADATA_ITEM 0) itemoff 16052 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 7 key (8733701521408 METADATA_ITEM 0) itemoff 16019 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 8 key (8733701537792 METADATA_ITEM 0) itemoff 15986 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 9 key (8733701554176 METADATA_ITEM 0) itemoff 15953 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 10 key (8733701570560 METADATA_ITEM 0) itemoff 15920 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 11 key (8733701586944 METADATA_ITEM 0) itemoff 15887 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 12 key (8733701603328 METADATA_ITEM 0) itemoff 15854 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 13 key (8733701619712 METADATA_ITEM 0) itemoff 15821 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 14 key (8733701636096 METADATA_ITEM 0) itemoff 15788 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 15 key (8733701652480 METADATA_ITEM 0) itemoff 15755 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 16 key (8733701668864 METADATA_ITEM 0) itemoff 15722 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 17 key (8733701685248 METADATA_ITEM 0) itemoff 15689 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 18 key (8733701701632 METADATA_ITEM 0) itemoff 15656 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 19 key (8733701718016 METADATA_ITEM 0) itemoff 15623 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 20 key (8733701734400 METADATA_ITEM 0) itemoff 15590 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 21 key (8733701750784 METADATA_ITEM 0) itemoff 15557 itemsize 33
		refs 1 gen 1584113 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 22 key (8733701783552 METADATA_ITEM 0) itemoff 15524 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 23 key (8733701799936 METADATA_ITEM 0) itemoff 15491 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 24 key (8733701816320 METADATA_ITEM 0) itemoff 15458 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 25 key (8733701832704 METADATA_ITEM 0) itemoff 15425 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 26 key (8733701865472 METADATA_ITEM 0) itemoff 15392 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 27 key (8733701881856 METADATA_ITEM 0) itemoff 15359 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 28 key (8733701898240 METADATA_ITEM 1) itemoff 15326 itemsize 33
		refs 1 gen 1474318 flags TREE_BLOCK
		tree block skinny level 1
		tree block backref root CSUM_TREE
	item 29 key (8733701914624 METADATA_ITEM 0) itemoff 15293 itemsize 33
		refs 1 gen 1474318 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 30 key (8733701931008 METADATA_ITEM 0) itemoff 15260 itemsize 33
		refs 1 gen 1480353 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 31 key (8733701947392 METADATA_ITEM 0) itemoff 15227 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 32 key (8733701963776 METADATA_ITEM 0) itemoff 15194 itemsize 33
		refs 1 gen 1480353 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 33 key (8733701980160 METADATA_ITEM 0) itemoff 15161 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 34 key (8733701996544 METADATA_ITEM 0) itemoff 15128 itemsize 33
		refs 1 gen 1581638 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 35 key (8733702012928 METADATA_ITEM 0) itemoff 15095 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 36 key (8733702029312 METADATA_ITEM 0) itemoff 15062 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 37 key (8733702045696 METADATA_ITEM 0) itemoff 15029 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 38 key (8733702062080 METADATA_ITEM 0) itemoff 14996 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 39 key (8733702078464 METADATA_ITEM 0) itemoff 14963 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 40 key (8733702094848 METADATA_ITEM 0) itemoff 14930 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 41 key (8733702111232 METADATA_ITEM 0) itemoff 14897 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 42 key (8733702127616 METADATA_ITEM 0) itemoff 14864 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 43 key (8733702144000 METADATA_ITEM 0) itemoff 14831 itemsize 33
		refs 1 gen 1581638 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 44 key (8733702176768 METADATA_ITEM 0) itemoff 14798 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 45 key (8733702193152 METADATA_ITEM 0) itemoff 14765 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 46 key (8733702209536 METADATA_ITEM 0) itemoff 14732 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 47 key (8733702225920 METADATA_ITEM 0) itemoff 14699 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 48 key (8733702242304 METADATA_ITEM 0) itemoff 14666 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 49 key (8733702258688 METADATA_ITEM 0) itemoff 14633 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 50 key (8733702275072 METADATA_ITEM 0) itemoff 14600 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 51 key (8733702291456 METADATA_ITEM 0) itemoff 14567 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 52 key (8733702307840 METADATA_ITEM 0) itemoff 14534 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 53 key (8733702324224 METADATA_ITEM 0) itemoff 14501 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 54 key (8733702340608 METADATA_ITEM 0) itemoff 14468 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 55 key (8733702356992 METADATA_ITEM 0) itemoff 14435 itemsize 33
		refs 1 gen 1360804 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 56 key (8733702389760 METADATA_ITEM 0) itemoff 14402 itemsize 33
		refs 1 gen 1360804 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 57 key (8733702406144 METADATA_ITEM 0) itemoff 14369 itemsize 33
		refs 1 gen 1360804 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 58 key (8733702422528 METADATA_ITEM 0) itemoff 14336 itemsize 33
		refs 1 gen 838551 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 59 key (8733702438912 METADATA_ITEM 0) itemoff 14303 itemsize 33
		refs 1 gen 1360804 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 60 key (8733702455296 METADATA_ITEM 0) itemoff 14270 itemsize 33
		refs 1 gen 1360804 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 61 key (8733702504448 METADATA_ITEM 0) itemoff 14237 itemsize 33
		refs 1 gen 300521 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 62 key (8733702520832 METADATA_ITEM 0) itemoff 14204 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 63 key (8733702537216 METADATA_ITEM 0) itemoff 14171 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 64 key (8733702553600 METADATA_ITEM 0) itemoff 14138 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 65 key (8733702569984 METADATA_ITEM 0) itemoff 14105 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 66 key (8733702586368 METADATA_ITEM 0) itemoff 14072 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 67 key (8733702602752 METADATA_ITEM 0) itemoff 14039 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 68 key (8733702619136 METADATA_ITEM 0) itemoff 14006 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 69 key (8733702635520 METADATA_ITEM 0) itemoff 13973 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 70 key (8733702651904 METADATA_ITEM 0) itemoff 13940 itemsize 33
		refs 1 gen 1602171 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 71 key (8733702668288 METADATA_ITEM 0) itemoff 13907 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 72 key (8733702684672 METADATA_ITEM 0) itemoff 13874 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 73 key (8733702701056 METADATA_ITEM 0) itemoff 13841 itemsize 33
		refs 1 gen 300259 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 74 key (8733702717440 METADATA_ITEM 0) itemoff 13808 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 75 key (8733702733824 METADATA_ITEM 0) itemoff 13775 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 76 key (8733702750208 METADATA_ITEM 0) itemoff 13742 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 77 key (8733702766592 METADATA_ITEM 0) itemoff 13709 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 78 key (8733702782976 METADATA_ITEM 0) itemoff 13676 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 79 key (8733702799360 METADATA_ITEM 0) itemoff 13643 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 80 key (8733702815744 METADATA_ITEM 0) itemoff 13610 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 81 key (8733702832128 METADATA_ITEM 0) itemoff 13577 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 82 key (8733702848512 METADATA_ITEM 0) itemoff 13544 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 83 key (8733702864896 METADATA_ITEM 0) itemoff 13511 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 84 key (8733702881280 METADATA_ITEM 0) itemoff 13478 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 85 key (8733702897664 METADATA_ITEM 0) itemoff 13445 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 86 key (8733702914048 METADATA_ITEM 0) itemoff 13412 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 87 key (8733702930432 METADATA_ITEM 0) itemoff 13379 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 88 key (8733702946816 METADATA_ITEM 0) itemoff 13346 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 89 key (8733702963200 METADATA_ITEM 0) itemoff 13313 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 90 key (8733702979584 METADATA_ITEM 0) itemoff 13280 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 91 key (8733702995968 METADATA_ITEM 0) itemoff 13247 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 92 key (8733703012352 METADATA_ITEM 0) itemoff 13214 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 93 key (8733703028736 METADATA_ITEM 0) itemoff 13181 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 94 key (8733703045120 METADATA_ITEM 0) itemoff 13148 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 95 key (8733703061504 METADATA_ITEM 0) itemoff 13115 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 96 key (8733703077888 METADATA_ITEM 0) itemoff 13082 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 97 key (8733703094272 METADATA_ITEM 0) itemoff 13049 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 98 key (8733703110656 METADATA_ITEM 0) itemoff 13016 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 99 key (8733703127040 METADATA_ITEM 0) itemoff 12983 itemsize 33
		refs 1 gen 3264 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 100 key (8733703143424 METADATA_ITEM 0) itemoff 12950 itemsize 33
		refs 1 gen 4166 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 101 key (8733703159808 METADATA_ITEM 0) itemoff 12917 itemsize 33
		refs 1 gen 1581638 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 102 key (8733703176192 METADATA_ITEM 0) itemoff 12884 itemsize 33
		refs 1 gen 1581638 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 103 key (8733703192576 METADATA_ITEM 0) itemoff 12851 itemsize 33
		refs 1 gen 1514029 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 104 key (8733703208960 METADATA_ITEM 0) itemoff 12818 itemsize 33
		refs 1 gen 4166 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 105 key (8733703225344 METADATA_ITEM 0) itemoff 12785 itemsize 33
		refs 1 gen 1514029 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 106 key (8733703241728 METADATA_ITEM 0) itemoff 12752 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 107 key (8733703258112 METADATA_ITEM 0) itemoff 12719 itemsize 33
		refs 1 gen 1581638 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 108 key (8733703274496 METADATA_ITEM 0) itemoff 12686 itemsize 33
		refs 1 gen 300521 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 109 key (8733703290880 METADATA_ITEM 0) itemoff 12653 itemsize 33
		refs 1 gen 1584112 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 110 key (8733703323648 METADATA_ITEM 0) itemoff 12620 itemsize 33
		refs 1 gen 1602171 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 111 key (8733703340032 METADATA_ITEM 0) itemoff 12587 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 112 key (8733703454720 METADATA_ITEM 0) itemoff 12554 itemsize 33
		refs 1 gen 1535773 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 113 key (8733703471104 METADATA_ITEM 0) itemoff 12521 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 114 key (8733703487488 METADATA_ITEM 0) itemoff 12488 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 115 key (8733703503872 METADATA_ITEM 0) itemoff 12455 itemsize 33
		refs 1 gen 519869 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 116 key (8733703520256 METADATA_ITEM 0) itemoff 12422 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 117 key (8733703536640 METADATA_ITEM 0) itemoff 12389 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 118 key (8733703553024 METADATA_ITEM 0) itemoff 12356 itemsize 33
		refs 1 gen 1538313 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 119 key (8733703569408 METADATA_ITEM 0) itemoff 12323 itemsize 33
		refs 1 gen 1602171 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 120 key (8733703602176 METADATA_ITEM 0) itemoff 12290 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 121 key (8733703651328 METADATA_ITEM 0) itemoff 12257 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 122 key (8733703733248 METADATA_ITEM 0) itemoff 12224 itemsize 33
		refs 1 gen 1510262 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 123 key (8733703749632 METADATA_ITEM 0) itemoff 12191 itemsize 33
		refs 1 gen 1601423 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 124 key (8733703766016 METADATA_ITEM 0) itemoff 12158 itemsize 33
		refs 1 gen 1509558 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 125 key (8733703798784 METADATA_ITEM 0) itemoff 12125 itemsize 33
		refs 1 gen 1480353 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 126 key (8733703815168 METADATA_ITEM 0) itemoff 12092 itemsize 33
		refs 1 gen 1480353 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 127 key (8733703831552 METADATA_ITEM 0) itemoff 12059 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 128 key (8733703847936 METADATA_ITEM 0) itemoff 12026 itemsize 33
		refs 1 gen 3256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 129 key (8733703864320 METADATA_ITEM 0) itemoff 11993 itemsize 33
		refs 1 gen 3246 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 130 key (8733703880704 METADATA_ITEM 0) itemoff 11960 itemsize 33
		refs 1 gen 3246 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 131 key (8733703897088 METADATA_ITEM 0) itemoff 11927 itemsize 33
		refs 1 gen 1518676 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 132 key (8733703913472 METADATA_ITEM 0) itemoff 11894 itemsize 33
		refs 1 gen 3246 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 133 key (8733703929856 METADATA_ITEM 0) itemoff 11861 itemsize 33
		refs 1 gen 3246 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 134 key (8733703946240 METADATA_ITEM 0) itemoff 11828 itemsize 33
		refs 1 gen 1590258 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root 11223
	item 135 key (8733703979008 METADATA_ITEM 0) itemoff 11795 itemsize 33
		refs 1 gen 749275 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 136 key (8733704060928 METADATA_ITEM 0) itemoff 11762 itemsize 33
		refs 1 gen 1509163 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 137 key (8733704126464 METADATA_ITEM 0) itemoff 11729 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 138 key (8733704142848 METADATA_ITEM 0) itemoff 11696 itemsize 33
		refs 1 gen 1496691 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 139 key (8733704159232 METADATA_ITEM 0) itemoff 11663 itemsize 33
		refs 1 gen 1496691 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 140 key (8733704241152 METADATA_ITEM 0) itemoff 11630 itemsize 33
		refs 1 gen 441944 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 141 key (8733704257536 METADATA_ITEM 0) itemoff 11597 itemsize 33
		refs 1 gen 1514029 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 142 key (8733704273920 METADATA_ITEM 0) itemoff 11564 itemsize 33
		refs 1 gen 904814 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 143 key (8733704290304 METADATA_ITEM 0) itemoff 11531 itemsize 33
		refs 1 gen 904814 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 144 key (8733704306688 METADATA_ITEM 0) itemoff 11498 itemsize 33
		refs 1 gen 904814 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 145 key (8733704323072 METADATA_ITEM 0) itemoff 11465 itemsize 33
		refs 1 gen 904814 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 146 key (8733704355840 METADATA_ITEM 0) itemoff 11432 itemsize 33
		refs 1 gen 1526001 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 147 key (8733704404992 METADATA_ITEM 0) itemoff 11399 itemsize 33
		refs 1 gen 300521 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 148 key (8733704421376 METADATA_ITEM 0) itemoff 11366 itemsize 33
		refs 1 gen 441944 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 149 key (8733704437760 METADATA_ITEM 0) itemoff 11333 itemsize 33
		refs 1 gen 441944 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 150 key (8733704454144 METADATA_ITEM 0) itemoff 11300 itemsize 33
		refs 1 gen 441944 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 151 key (8733704470528 METADATA_ITEM 0) itemoff 11267 itemsize 33
		refs 1 gen 441944 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 152 key (8733704519680 METADATA_ITEM 0) itemoff 11234 itemsize 33
		refs 1 gen 1497583 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 153 key (8733704536064 METADATA_ITEM 0) itemoff 11201 itemsize 33
		refs 1 gen 441944 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 154 key (8733704617984 METADATA_ITEM 0) itemoff 11168 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 155 key (8733704667136 METADATA_ITEM 0) itemoff 11135 itemsize 33
		refs 1 gen 441944 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 156 key (8733704683520 METADATA_ITEM 0) itemoff 11102 itemsize 33
		refs 1 gen 300521 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 157 key (8733704716288 METADATA_ITEM 0) itemoff 11069 itemsize 33
		refs 1 gen 4166 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 158 key (8733704765440 METADATA_ITEM 0) itemoff 11036 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 159 key (8733704798208 METADATA_ITEM 0) itemoff 11003 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 160 key (8733704847360 METADATA_ITEM 0) itemoff 10970 itemsize 33
		refs 1 gen 115450 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 161 key (8733704863744 METADATA_ITEM 0) itemoff 10937 itemsize 33
		refs 1 gen 115450 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 162 key (8733704880128 METADATA_ITEM 0) itemoff 10904 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 163 key (8733704912896 METADATA_ITEM 0) itemoff 10871 itemsize 33
		refs 1 gen 115450 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 164 key (8733704945664 METADATA_ITEM 0) itemoff 10838 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 165 key (8733704962048 METADATA_ITEM 0) itemoff 10805 itemsize 33
		refs 1 gen 115450 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 166 key (8733704978432 METADATA_ITEM 0) itemoff 10772 itemsize 33
		refs 1 gen 736618 flags TREE_BLOCK
		tree block skinny level 0
		shared block backref parent 10679198629888
	item 167 key (8733705043968 METADATA_ITEM 0) itemoff 10739 itemsize 33
		refs 1 gen 1554576 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 168 key (8733705093120 METADATA_ITEM 0) itemoff 10706 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 169 key (8733705109504 METADATA_ITEM 0) itemoff 10673 itemsize 33
		refs 1 gen 1554576 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 170 key (8733705125888 METADATA_ITEM 0) itemoff 10640 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 171 key (8733705142272 METADATA_ITEM 0) itemoff 10607 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 172 key (8733705175040 METADATA_ITEM 0) itemoff 10574 itemsize 33
		refs 1 gen 3221 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 173 key (8733705191424 METADATA_ITEM 0) itemoff 10541 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 174 key (8733705207808 METADATA_ITEM 0) itemoff 10508 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 175 key (8733705224192 METADATA_ITEM 0) itemoff 10475 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 176 key (8733705240576 METADATA_ITEM 0) itemoff 10442 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 177 key (8733705256960 METADATA_ITEM 0) itemoff 10409 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 178 key (8733705273344 METADATA_ITEM 0) itemoff 10376 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 179 key (8733705289728 METADATA_ITEM 0) itemoff 10343 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 180 key (8733705306112 METADATA_ITEM 0) itemoff 10310 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 181 key (8733705322496 METADATA_ITEM 0) itemoff 10277 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 182 key (8733705338880 METADATA_ITEM 0) itemoff 10244 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 183 key (8733705453568 METADATA_ITEM 0) itemoff 10211 itemsize 33
		refs 1 gen 68949 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 184 key (8733705469952 METADATA_ITEM 0) itemoff 10178 itemsize 33
		refs 1 gen 904828 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 185 key (8733705502720 METADATA_ITEM 0) itemoff 10145 itemsize 33
		refs 1 gen 1530500 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 186 key (8733705535488 METADATA_ITEM 0) itemoff 10112 itemsize 33
		refs 1 gen 1602171 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 187 key (8733705551872 METADATA_ITEM 0) itemoff 10079 itemsize 33
		refs 1 gen 904814 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 188 key (8733705568256 METADATA_ITEM 0) itemoff 10046 itemsize 33
		refs 1 gen 904814 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 189 key (8733705584640 METADATA_ITEM 0) itemoff 10013 itemsize 33
		refs 1 gen 1530500 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 190 key (8733705601024 METADATA_ITEM 0) itemoff 9980 itemsize 33
		refs 1 gen 1522822 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 191 key (8733705633792 METADATA_ITEM 0) itemoff 9947 itemsize 33
		refs 1 gen 3846 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 192 key (8733705650176 METADATA_ITEM 0) itemoff 9914 itemsize 33
		refs 1 gen 1522822 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 193 key (8733705666560 METADATA_ITEM 0) itemoff 9881 itemsize 33
		refs 1 gen 1522822 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 194 key (8733705682944 METADATA_ITEM 0) itemoff 9848 itemsize 33
		refs 1 gen 904828 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 195 key (8733705699328 METADATA_ITEM 0) itemoff 9815 itemsize 33
		refs 1 gen 1514029 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 196 key (8733705715712 METADATA_ITEM 0) itemoff 9782 itemsize 33
		refs 1 gen 1522822 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m K.s./btrfsfinspect-internaludump-treeb-b 313577775284224 /dev/mapper/dshelf1a
btrfs-progs v5.16.2
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
FS_INFO IS 0x56312cc84870
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x56312cc84870
leaf 13577775284224 items 102 free space 8215 generation 1602088 owner FS_TREE
leaf 13577775284224 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
	item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
		generation 3 transid 1602088 size 4638 nbytes 16384
		block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
		sequence 152572 flags 0x0(none)
		atime 1518920225.99299649 (2018-02-17 18:17:05)
		ctime 1648458062.674404554 (2022-03-28 02:01:02)
		mtime 1648458062.674404554 (2022-03-28 02:01:02)
		otime 1518411216.0 (2018-02-11 20:53:36)
	item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12
		index 0 namelen 2 name: ..
	item 2 key (256 DIR_ITEM 35173225) itemoff 16055 itemsize 56
		location key (165295 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601935 data_len 0 name_len 26
		name: other_ro.20220327_00:24:58
	item 3 key (256 DIR_ITEM 37377682) itemoff 15995 itemsize 60
		location key (165387 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602062 data_len 0 name_len 30
		name: other_hourly.20220327_22:01:01
	item 4 key (256 DIR_ITEM 65985264) itemoff 15939 itemsize 56
		location key (164623 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601065 data_len 0 name_len 26
		name: Sound_ro.20220320_00:39:38
	item 5 key (256 DIR_ITEM 121085187) itemoff 15885 itemsize 54
		location key (164633 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601080 data_len 0 name_len 24
		name: Win_ro.20220320_01:12:38
	item 6 key (256 DIR_ITEM 122346812) itemoff 15829 itemsize 56
		location key (163920 ROOT_ITEM 18446744073709551615) type DIR
		transid 1599635 data_len 0 name_len 26
		name: Video_ro.20220313_00:38:01
	item 7 key (256 DIR_ITEM 167852107) itemoff 15769 itemsize 60
		location key (159786 ROOT_ITEM 18446744073709551615) type DIR
		transid 1591390 data_len 0 name_len 30
		name: Sound_weekly.20220130_02:03:01
	item 8 key (256 DIR_ITEM 185979643) itemoff 15714 itemsize 55
		location key (164622 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601062 data_len 0 name_len 25
		name: Soft_ro.20220320_00:35:42
	item 9 key (256 DIR_ITEM 188444463) itemoff 15655 itemsize 59
		location key (165400 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602080 data_len 0 name_len 29
		name: Video_daily.20220328_01:02:01
	item 10 key (256 DIR_ITEM 222467894) itemoff 15596 itemsize 59
		location key (165100 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601682 data_len 0 name_len 29
		name: media_daily.20220325_01:02:01
	item 11 key (256 DIR_ITEM 276733595) itemoff 15537 itemsize 59
		location key (165307 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601959 data_len 0 name_len 29
		name: other_daily.20220327_01:02:01
	item 12 key (256 DIR_ITEM 341062093) itemoff 15481 itemsize 56
		location key (163917 ROOT_ITEM 18446744073709551615) type DIR
		transid 1599627 data_len 0 name_len 26
		name: other_ro.20220313_00:26:11
	item 13 key (256 DIR_ITEM 350762628) itemoff 15425 itemsize 56
		location key (162632 ROOT_ITEM 18446744073709551615) type DIR
		transid 1596353 data_len 0 name_len 26
		name: Video_ro.20220227_00:46:32
	item 14 key (256 DIR_ITEM 367443219) itemoff 15365 itemsize 60
		location key (165399 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602078 data_len 0 name_len 30
		name: other_hourly.20220328_01:01:01
	item 15 key (256 DIR_ITEM 392444674) itemoff 15326 itemsize 39
		location key (103481 ROOT_ITEM 18446744073709551615) type DIR
		transid 1455725 data_len 0 name_len 9
		name: .beeshome
	item 16 key (256 DIR_ITEM 597156124) itemoff 15266 itemsize 60
		location key (165398 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602077 data_len 0 name_len 30
		name: media_hourly.20220328_01:01:01
	item 17 key (256 DIR_ITEM 640492716) itemoff 15206 itemsize 60
		location key (165396 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602075 data_len 0 name_len 30
		name: Video_hourly.20220328_01:01:01
	item 18 key (256 DIR_ITEM 668082479) itemoff 15168 itemsize 38
		location key (1496 INODE_ITEM 0) type SYMLINK
		transid 1601950 data_len 0 name_len 8
		name: Win_last
	item 19 key (256 DIR_ITEM 713697142) itemoff 15108 itemsize 60
		location key (161198 ROOT_ITEM 18446744073709551615) type DIR
		transid 1594244 data_len 0 name_len 30
		name: Sound_weekly.20220213_02:03:01
	item 20 key (256 DIR_ITEM 725111845) itemoff 15053 itemsize 55
		location key (163918 ROOT_ITEM 18446744073709551615) type DIR
		transid 1599629 data_len 0 name_len 25
		name: Soft_ro.20220313_00:33:42
	item 21 key (256 DIR_ITEM 769653260) itemoff 14994 itemsize 59
		location key (165403 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602083 data_len 0 name_len 29
		name: other_daily.20220328_01:02:01
	item 22 key (256 DIR_ITEM 916065208) itemoff 14935 itemsize 59
		location key (165304 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601956 data_len 0 name_len 29
		name: Video_daily.20220327_01:02:01
	item 23 key (256 DIR_ITEM 971435966) itemoff 14879 itemsize 56
		location key (163916 ROOT_ITEM 18446744073709551615) type DIR
		transid 1599626 data_len 0 name_len 26
		name: media_ro.20220313_00:24:44
	item 24 key (256 DIR_ITEM 981009741) itemoff 14846 itemsize 33
		location key (11224 ROOT_ITEM 18446744073709551615) type DIR
		transid 4062 data_len 0 name_len 3
		name: Win
	item 25 key (256 DIR_ITEM 1200370785) itemoff 14786 itemsize 60
		location key (165405 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602086 data_len 0 name_len 30
		name: Sound_hourly.20220328_02:01:01
	item 26 key (256 DIR_ITEM 1268043220) itemoff 14726 itemsize 60
		location key (165394 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602072 data_len 0 name_len 30
		name: media_hourly.20220328_00:01:01
	item 27 key (256 DIR_ITEM 1288340042) itemoff 14691 itemsize 35
		location key (11222 ROOT_ITEM 18446744073709551615) type DIR
		transid 4058 data_len 0 name_len 5
		name: media
	item 28 key (256 DIR_ITEM 1311639652) itemoff 14631 itemsize 60
		location key (165392 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602070 data_len 0 name_len 30
		name: Video_hourly.20220328_00:01:01
	item 29 key (256 DIR_ITEM 1388422445) itemoff 14575 itemsize 56
		location key (162628 ROOT_ITEM 18446744073709551615) type DIR
		transid 1596345 data_len 0 name_len 26
		name: media_ro.20220227_00:24:39
	item 30 key (256 DIR_ITEM 1431596328) itemoff 14516 itemsize 59
		location key (165200 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601811 data_len 0 name_len 29
		name: media_daily.20220326_01:02:01
	item 31 key (256 DIR_ITEM 1438703253) itemoff 14456 itemsize 60
		location key (160495 ROOT_ITEM 18446744073709551615) type DIR
		transid 1592683 data_len 0 name_len 30
		name: Sound_weekly.20220206_02:03:01
	item 32 key (256 DIR_ITEM 1440530326) itemoff 14423 itemsize 33
		location key (1003 INODE_ITEM 0) type FILE
		transid 1448943 data_len 0 name_len 3
		name: DS1
	item 33 key (256 DIR_ITEM 1476660117) itemoff 14364 itemsize 59
		location key (165401 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602081 data_len 0 name_len 29
		name: Sound_daily.20220328_01:02:01
	item 34 key (256 DIR_ITEM 1509090277) itemoff 14304 itemsize 60
		location key (165388 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602064 data_len 0 name_len 30
		name: Video_hourly.20220327_23:01:01
	item 35 key (256 DIR_ITEM 1516701308) itemoff 14250 itemsize 54
		location key (165299 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601947 data_len 0 name_len 24
		name: Win_ro.20220327_00:37:15
	item 36 key (256 DIR_ITEM 1548232277) itemoff 14190 itemsize 60
		location key (165390 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602066 data_len 0 name_len 30
		name: media_hourly.20220327_23:01:01
	item 37 key (256 DIR_ITEM 1578913405) itemoff 14155 itemsize 35
		location key (11322 ROOT_ITEM 18446744073709551615) type DIR
		transid 4223 data_len 0 name_len 5
		name: Sound
	item 38 key (256 DIR_ITEM 1597000160) itemoff 14115 itemsize 40
		location key (1494 INODE_ITEM 0) type SYMLINK
		transid 1601944 data_len 0 name_len 10
		name: Sound_last
	item 39 key (256 DIR_ITEM 1689324197) itemoff 14055 itemsize 60
		location key (163317 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597264 data_len 0 name_len 30
		name: Sound_weekly.20220306_02:03:01
	item 40 key (256 DIR_ITEM 1705340674) itemoff 13996 itemsize 59
		location key (165305 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601957 data_len 0 name_len 29
		name: Sound_daily.20220327_01:02:01
	item 41 key (256 DIR_ITEM 1731806220) itemoff 13940 itemsize 56
		location key (165298 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601944 data_len 0 name_len 26
		name: Video_ro.20220327_00:35:50
	item 42 key (256 DIR_ITEM 1782151770) itemoff 13880 itemsize 60
		location key (165391 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602067 data_len 0 name_len 30
		name: other_hourly.20220327_23:01:01
	item 43 key (256 DIR_ITEM 1808375180) itemoff 13824 itemsize 56
		location key (163301 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597239 data_len 0 name_len 26
		name: Sound_ro.20220306_00:45:46
	item 44 key (256 DIR_ITEM 1925286825) itemoff 13768 itemsize 56
		location key (164621 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601060 data_len 0 name_len 26
		name: other_ro.20220320_00:25:35
	item 45 key (256 DIR_ITEM 2030955372) itemoff 13712 itemsize 56
		location key (163919 ROOT_ITEM 18446744073709551615) type DIR
		transid 1599632 data_len 0 name_len 26
		name: Sound_ro.20220313_00:37:25
	item 46 key (256 DIR_ITEM 2112197083) itemoff 13652 itemsize 60
		location key (165395 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602073 data_len 0 name_len 30
		name: other_hourly.20220328_00:01:01
	item 47 key (256 DIR_ITEM 2127973047) itemoff 13617 itemsize 35
		location key (11221 ROOT_ITEM 18446744073709551615) type DIR
		transid 4042 data_len 0 name_len 5
		name: other
	item 48 key (256 DIR_ITEM 2146234148) itemoff 13583 itemsize 34
		location key (11223 ROOT_ITEM 18446744073709551615) type DIR
		transid 4061 data_len 0 name_len 4
		name: Soft
	item 49 key (256 DIR_ITEM 2153154160) itemoff 13523 itemsize 60
		location key (165389 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602065 data_len 0 name_len 30
		name: Sound_hourly.20220327_23:01:01
	item 50 key (256 DIR_ITEM 2201886496) itemoff 13464 itemsize 59
		location key (165101 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601683 data_len 0 name_len 29
		name: other_daily.20220325_01:02:01
	item 51 key (256 DIR_ITEM 2279840979) itemoff 13408 itemsize 56
		location key (162629 ROOT_ITEM 18446744073709551615) type DIR
		transid 1596346 data_len 0 name_len 26
		name: other_ro.20220227_00:25:38
	item 52 key (256 DIR_ITEM 2313060016) itemoff 13348 itemsize 60
		location key (160496 ROOT_ITEM 18446744073709551615) type DIR
		transid 1592684 data_len 0 name_len 30
		name: media_weekly.20220206_02:03:01
	item 53 key (256 DIR_ITEM 2354277240) itemoff 13308 itemsize 40
		location key (1492 INODE_ITEM 0) type SYMLINK
		transid 1601938 data_len 0 name_len 10
		name: other_last
	item 54 key (256 DIR_ITEM 2355411712) itemoff 13248 itemsize 60
		location key (160494 ROOT_ITEM 18446744073709551615) type DIR
		transid 1592682 data_len 0 name_len 30
		name: Video_weekly.20220206_02:03:01
	item 55 key (256 DIR_ITEM 2377365299) itemoff 13192 itemsize 56
		location key (165297 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601941 data_len 0 name_len 26
		name: Sound_ro.20220327_00:35:22
	item 56 key (256 DIR_ITEM 2396976783) itemoff 13132 itemsize 60
		location key (163319 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597266 data_len 0 name_len 30
		name: other_weekly.20220306_02:03:01
	item 57 key (256 DIR_ITEM 2456187574) itemoff 13076 itemsize 56
		location key (163298 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597234 data_len 0 name_len 26
		name: media_ro.20220306_00:24:43
	item 58 key (256 DIR_ITEM 2467975514) itemoff 13021 itemsize 55
		location key (163300 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597237 data_len 0 name_len 25
		name: Soft_ro.20220306_00:33:55
	item 59 key (256 DIR_ITEM 2542431729) itemoff 12961 itemsize 60
		location key (165393 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602071 data_len 0 name_len 30
		name: Sound_hourly.20220328_00:01:01
	item 60 key (256 DIR_ITEM 2610074692) itemoff 12901 itemsize 60
		location key (165406 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602087 data_len 0 name_len 30
		name: media_hourly.20220328_02:01:01
	item 61 key (256 DIR_ITEM 2650936973) itemoff 12842 itemsize 59
		location key (165306 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601958 data_len 0 name_len 29
		name: media_daily.20220327_01:02:01
	item 62 key (256 DIR_ITEM 2653442548) itemoff 12782 itemsize 60
		location key (165404 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602085 data_len 0 name_len 30
		name: Video_hourly.20220328_02:01:01
	item 63 key (256 DIR_ITEM 2717852207) itemoff 12728 itemsize 54
		location key (163921 ROOT_ITEM 18446744073709551615) type DIR
		transid 1599637 data_len 0 name_len 24
		name: Win_ro.20220313_00:38:28
	item 64 key (256 DIR_ITEM 2745397786) itemoff 12669 itemsize 59
		location key (165402 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602082 data_len 0 name_len 29
		name: media_daily.20220328_01:02:01
	item 65 key (256 DIR_ITEM 2782562819) itemoff 12610 itemsize 59
		location key (165098 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601680 data_len 0 name_len 29
		name: Video_daily.20220325_01:02:01
	item 66 key (256 DIR_ITEM 2793356223) itemoff 12554 itemsize 56
		location key (163299 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597235 data_len 0 name_len 26
		name: other_ro.20220306_00:25:40
	item 67 key (256 DIR_ITEM 2811536125) itemoff 12519 itemsize 35
		location key (11288 ROOT_ITEM 18446744073709551615) type DIR
		transid 4174 data_len 0 name_len 5
		name: Video
	item 68 key (256 DIR_ITEM 2827986674) itemoff 12485 itemsize 34
		location key (1010 INODE_ITEM 0) type FILE
		transid 1455723 data_len 0 name_len 4
		name: copy
	item 69 key (256 DIR_ITEM 2917386315) itemoff 12425 itemsize 60
		location key (165407 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602088 data_len 0 name_len 30
		name: other_hourly.20220328_02:01:01
	item 70 key (256 DIR_ITEM 2935166119) itemoff 12366 itemsize 59
		location key (165199 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601810 data_len 0 name_len 29
		name: Sound_daily.20220326_01:02:01
	item 71 key (256 DIR_ITEM 2935255735) itemoff 12311 itemsize 55
		location key (165296 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601938 data_len 0 name_len 25
		name: Soft_ro.20220327_00:32:46
	item 72 key (256 DIR_ITEM 3098520192) itemoff 12251 itemsize 60
		location key (163318 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597265 data_len 0 name_len 30
		name: media_weekly.20220306_02:03:01
	item 73 key (256 DIR_ITEM 3170169652) itemoff 12195 itemsize 56
		location key (164624 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601068 data_len 0 name_len 26
		name: Video_ro.20220320_00:40:27
	item 74 key (256 DIR_ITEM 3172298544) itemoff 12135 itemsize 60
		location key (163316 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597263 data_len 0 name_len 30
		name: Video_weekly.20220306_02:03:01
	item 75 key (256 DIR_ITEM 3215925951) itemoff 12075 itemsize 60
		location key (160497 ROOT_ITEM 18446744073709551615) type DIR
		transid 1592685 data_len 0 name_len 30
		name: other_weekly.20220206_02:03:01
	item 76 key (256 DIR_ITEM 3236289372) itemoff 12015 itemsize 60
		location key (161200 ROOT_ITEM 18446744073709551615) type DIR
		transid 1594246 data_len 0 name_len 30
		name: other_weekly.20220213_02:03:01
	item 77 key (256 DIR_ITEM 3290748772) itemoff 11959 itemsize 56
		location key (163302 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597242 data_len 0 name_len 26
		name: Video_ro.20220306_00:46:14
	item 78 key (256 DIR_ITEM 3360445642) itemoff 11903 itemsize 56
		location key (162631 ROOT_ITEM 18446744073709551615) type DIR
		transid 1596351 data_len 0 name_len 26
		name: Sound_ro.20220227_00:40:41
	item 79 key (256 DIR_ITEM 3470690534) itemoff 11847 itemsize 56
		location key (164620 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601059 data_len 0 name_len 26
		name: media_ro.20220320_00:24:47
	item 80 key (256 DIR_ITEM 3550830046) itemoff 11787 itemsize 60
		location key (159785 ROOT_ITEM 18446744073709551615) type DIR
		transid 1591389 data_len 0 name_len 30
		name: Video_weekly.20220130_02:03:01
	item 81 key (256 DIR_ITEM 3592394862) itemoff 11727 itemsize 60
		location key (159787 ROOT_ITEM 18446744073709551615) type DIR
		transid 1591391 data_len 0 name_len 30
		name: media_weekly.20220130_02:03:01
	item 82 key (256 DIR_ITEM 3676890430) itemoff 11668 itemsize 59
		location key (165201 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601812 data_len 0 name_len 29
		name: other_daily.20220326_01:02:01
	item 83 key (256 DIR_ITEM 3765355617) itemoff 11608 itemsize 60
		location key (159788 ROOT_ITEM 18446744073709551615) type DIR
		transid 1591392 data_len 0 name_len 30
		name: other_weekly.20220130_02:03:01
	item 84 key (256 DIR_ITEM 3787964139) itemoff 11554 itemsize 54
		location key (163303 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597244 data_len 0 name_len 24
		name: Win_ro.20220306_00:47:32
	item 85 key (256 DIR_ITEM 3799902853) itemoff 11514 itemsize 40
		location key (1491 INODE_ITEM 0) type SYMLINK
		transid 1601935 data_len 0 name_len 10
		name: media_last
	item 86 key (256 DIR_ITEM 3872736471) itemoff 11460 itemsize 54
		location key (162645 ROOT_ITEM 18446744073709551615) type DIR
		transid 1596372 data_len 0 name_len 24
		name: Win_ro.20220227_02:10:25
	item 87 key (256 DIR_ITEM 3951915213) itemoff 11420 itemsize 40
		location key (1495 INODE_ITEM 0) type SYMLINK
		transid 1601947 data_len 0 name_len 10
		name: Video_last
	item 88 key (256 DIR_ITEM 4079895267) itemoff 11360 itemsize 60
		location key (161197 ROOT_ITEM 18446744073709551615) type DIR
		transid 1594243 data_len 0 name_len 30
		name: Video_weekly.20220213_02:03:01
	item 89 key (256 DIR_ITEM 4084751325) itemoff 11321 itemsize 39
		location key (1493 INODE_ITEM 0) type SYMLINK
		transid 1601941 data_len 0 name_len 9
		name: Soft_last
	item 90 key (256 DIR_ITEM 4136929107) itemoff 11261 itemsize 60
		location key (161199 ROOT_ITEM 18446744073709551615) type DIR
		transid 1594245 data_len 0 name_len 30
		name: media_weekly.20220213_02:03:01
	item 91 key (256 DIR_ITEM 4142263993) itemoff 11202 itemsize 59
		location key (165099 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601681 data_len 0 name_len 29
		name: Sound_daily.20220325_01:02:01
	item 92 key (256 DIR_ITEM 4156034002) itemoff 11147 itemsize 55
		location key (162630 ROOT_ITEM 18446744073709551615) type DIR
		transid 1596348 data_len 0 name_len 25
		name: Soft_ro.20220227_00:34:48
	item 93 key (256 DIR_ITEM 4198095398) itemoff 11091 itemsize 56
		location key (165294 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601934 data_len 0 name_len 26
		name: media_ro.20220327_00:24:32
	item 94 key (256 DIR_ITEM 4258033693) itemoff 11032 itemsize 59
		location key (165198 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601809 data_len 0 name_len 29
		name: Video_daily.20220326_01:02:01
	item 95 key (256 DIR_ITEM 4287185209) itemoff 10972 itemsize 60
		location key (165397 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602076 data_len 0 name_len 30
		name: Sound_hourly.20220328_01:01:01
	item 96 key (256 DIR_INDEX 8) itemoff 10937 itemsize 35
		location key (11221 ROOT_ITEM 18446744073709551615) type DIR
		transid 4042 data_len 0 name_len 5
		name: other
	item 97 key (256 DIR_INDEX 9) itemoff 10902 itemsize 35
		location key (11222 ROOT_ITEM 18446744073709551615) type DIR
		transid 4058 data_len 0 name_len 5
		name: media
	item 98 key (256 DIR_INDEX 10) itemoff 10868 itemsize 34
		location key (11223 ROOT_ITEM 18446744073709551615) type DIR
		transid 4061 data_len 0 name_len 4
		name: Soft
	item 99 key (256 DIR_INDEX 11) itemoff 10835 itemsize 33
		location key (11224 ROOT_ITEM 18446744073709551615) type DIR
		transid 4062 data_len 0 name_len 3
		name: Win
	item 100 key (256 DIR_INDEX 17) itemoff 10800 itemsize 35
		location key (11288 ROOT_ITEM 18446744073709551615) type DIR
		transid 4174 data_len 0 name_len 5
		name: Video
	item 101 key (256 DIR_INDEX 51) itemoff 10765 itemsize 35
		location key (11322 ROOT_ITEM 18446744073709551615) type DIR
		transid 4223 data_len 0 name_len 5
		name: Sound
1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m K.s./btrfsfinspect-internaludump-tree -b 313577775284224 /dev/mapper/dshelf1a
btrfs-progs v5.16.2
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
FS_INFO IS 0x55fb3b29e870
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x55fb3b29e870
leaf 13577775284224 items 102 free space 8215 generation 1602088 owner FS_TREE
leaf 13577775284224 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
	item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
		generation 3 transid 1602088 size 4638 nbytes 16384
		block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
		sequence 152572 flags 0x0(none)
		atime 1518920225.99299649 (2018-02-17 18:17:05)
		ctime 1648458062.674404554 (2022-03-28 02:01:02)
		mtime 1648458062.674404554 (2022-03-28 02:01:02)
		otime 1518411216.0 (2018-02-11 20:53:36)
	item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12
		index 0 namelen 2 name: ..
	item 2 key (256 DIR_ITEM 35173225) itemoff 16055 itemsize 56
		location key (165295 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601935 data_len 0 name_len 26
		name: other_ro.20220327_00:24:58
	item 3 key (256 DIR_ITEM 37377682) itemoff 15995 itemsize 60
		location key (165387 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602062 data_len 0 name_len 30
		name: other_hourly.20220327_22:01:01
	item 4 key (256 DIR_ITEM 65985264) itemoff 15939 itemsize 56
		location key (164623 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601065 data_len 0 name_len 26
		name: Sound_ro.20220320_00:39:38
	item 5 key (256 DIR_ITEM 121085187) itemoff 15885 itemsize 54
		location key (164633 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601080 data_len 0 name_len 24
		name: Win_ro.20220320_01:12:38
	item 6 key (256 DIR_ITEM 122346812) itemoff 15829 itemsize 56
		location key (163920 ROOT_ITEM 18446744073709551615) type DIR
		transid 1599635 data_len 0 name_len 26
		name: Video_ro.20220313_00:38:01
	item 7 key (256 DIR_ITEM 167852107) itemoff 15769 itemsize 60
		location key (159786 ROOT_ITEM 18446744073709551615) type DIR
		transid 1591390 data_len 0 name_len 30
		name: Sound_weekly.20220130_02:03:01
	item 8 key (256 DIR_ITEM 185979643) itemoff 15714 itemsize 55
		location key (164622 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601062 data_len 0 name_len 25
		name: Soft_ro.20220320_00:35:42
	item 9 key (256 DIR_ITEM 188444463) itemoff 15655 itemsize 59
		location key (165400 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602080 data_len 0 name_len 29
		name: Video_daily.20220328_01:02:01
	item 10 key (256 DIR_ITEM 222467894) itemoff 15596 itemsize 59
		location key (165100 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601682 data_len 0 name_len 29
		name: media_daily.20220325_01:02:01
	item 11 key (256 DIR_ITEM 276733595) itemoff 15537 itemsize 59
		location key (165307 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601959 data_len 0 name_len 29
		name: other_daily.20220327_01:02:01
	item 12 key (256 DIR_ITEM 341062093) itemoff 15481 itemsize 56
		location key (163917 ROOT_ITEM 18446744073709551615) type DIR
		transid 1599627 data_len 0 name_len 26
		name: other_ro.20220313_00:26:11
	item 13 key (256 DIR_ITEM 350762628) itemoff 15425 itemsize 56
		location key (162632 ROOT_ITEM 18446744073709551615) type DIR
		transid 1596353 data_len 0 name_len 26
		name: Video_ro.20220227_00:46:32
	item 14 key (256 DIR_ITEM 367443219) itemoff 15365 itemsize 60
		location key (165399 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602078 data_len 0 name_len 30
		name: other_hourly.20220328_01:01:01
	item 15 key (256 DIR_ITEM 392444674) itemoff 15326 itemsize 39
		location key (103481 ROOT_ITEM 18446744073709551615) type DIR
		transid 1455725 data_len 0 name_len 9
		name: .beeshome
	item 16 key (256 DIR_ITEM 597156124) itemoff 15266 itemsize 60
		location key (165398 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602077 data_len 0 name_len 30
		name: media_hourly.20220328_01:01:01
	item 17 key (256 DIR_ITEM 640492716) itemoff 15206 itemsize 60
		location key (165396 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602075 data_len 0 name_len 30
		name: Video_hourly.20220328_01:01:01
	item 18 key (256 DIR_ITEM 668082479) itemoff 15168 itemsize 38
		location key (1496 INODE_ITEM 0) type SYMLINK
		transid 1601950 data_len 0 name_len 8
		name: Win_last
	item 19 key (256 DIR_ITEM 713697142) itemoff 15108 itemsize 60
		location key (161198 ROOT_ITEM 18446744073709551615) type DIR
		transid 1594244 data_len 0 name_len 30
		name: Sound_weekly.20220213_02:03:01
	item 20 key (256 DIR_ITEM 725111845) itemoff 15053 itemsize 55
		location key (163918 ROOT_ITEM 18446744073709551615) type DIR
		transid 1599629 data_len 0 name_len 25
		name: Soft_ro.20220313_00:33:42
	item 21 key (256 DIR_ITEM 769653260) itemoff 14994 itemsize 59
		location key (165403 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602083 data_len 0 name_len 29
		name: other_daily.20220328_01:02:01
	item 22 key (256 DIR_ITEM 916065208) itemoff 14935 itemsize 59
		location key (165304 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601956 data_len 0 name_len 29
		name: Video_daily.20220327_01:02:01
	item 23 key (256 DIR_ITEM 971435966) itemoff 14879 itemsize 56
		location key (163916 ROOT_ITEM 18446744073709551615) type DIR
		transid 1599626 data_len 0 name_len 26
		name: media_ro.20220313_00:24:44
	item 24 key (256 DIR_ITEM 981009741) itemoff 14846 itemsize 33
		location key (11224 ROOT_ITEM 18446744073709551615) type DIR
		transid 4062 data_len 0 name_len 3
		name: Win
	item 25 key (256 DIR_ITEM 1200370785) itemoff 14786 itemsize 60
		location key (165405 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602086 data_len 0 name_len 30
		name: Sound_hourly.20220328_02:01:01
	item 26 key (256 DIR_ITEM 1268043220) itemoff 14726 itemsize 60
		location key (165394 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602072 data_len 0 name_len 30
		name: media_hourly.20220328_00:01:01
	item 27 key (256 DIR_ITEM 1288340042) itemoff 14691 itemsize 35
		location key (11222 ROOT_ITEM 18446744073709551615) type DIR
		transid 4058 data_len 0 name_len 5
		name: media
	item 28 key (256 DIR_ITEM 1311639652) itemoff 14631 itemsize 60
		location key (165392 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602070 data_len 0 name_len 30
		name: Video_hourly.20220328_00:01:01
	item 29 key (256 DIR_ITEM 1388422445) itemoff 14575 itemsize 56
		location key (162628 ROOT_ITEM 18446744073709551615) type DIR
		transid 1596345 data_len 0 name_len 26
		name: media_ro.20220227_00:24:39
	item 30 key (256 DIR_ITEM 1431596328) itemoff 14516 itemsize 59
		location key (165200 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601811 data_len 0 name_len 29
		name: media_daily.20220326_01:02:01
	item 31 key (256 DIR_ITEM 1438703253) itemoff 14456 itemsize 60
		location key (160495 ROOT_ITEM 18446744073709551615) type DIR
		transid 1592683 data_len 0 name_len 30
		name: Sound_weekly.20220206_02:03:01
	item 32 key (256 DIR_ITEM 1440530326) itemoff 14423 itemsize 33
		location key (1003 INODE_ITEM 0) type FILE
		transid 1448943 data_len 0 name_len 3
		name: DS1
	item 33 key (256 DIR_ITEM 1476660117) itemoff 14364 itemsize 59
		location key (165401 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602081 data_len 0 name_len 29
		name: Sound_daily.20220328_01:02:01
	item 34 key (256 DIR_ITEM 1509090277) itemoff 14304 itemsize 60
		location key (165388 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602064 data_len 0 name_len 30
		name: Video_hourly.20220327_23:01:01
	item 35 key (256 DIR_ITEM 1516701308) itemoff 14250 itemsize 54
		location key (165299 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601947 data_len 0 name_len 24
		name: Win_ro.20220327_00:37:15
	item 36 key (256 DIR_ITEM 1548232277) itemoff 14190 itemsize 60
		location key (165390 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602066 data_len 0 name_len 30
		name: media_hourly.20220327_23:01:01
	item 37 key (256 DIR_ITEM 1578913405) itemoff 14155 itemsize 35
		location key (11322 ROOT_ITEM 18446744073709551615) type DIR
		transid 4223 data_len 0 name_len 5
		name: Sound
	item 38 key (256 DIR_ITEM 1597000160) itemoff 14115 itemsize 40
		location key (1494 INODE_ITEM 0) type SYMLINK
		transid 1601944 data_len 0 name_len 10
		name: Sound_last
	item 39 key (256 DIR_ITEM 1689324197) itemoff 14055 itemsize 60
		location key (163317 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597264 data_len 0 name_len 30
		name: Sound_weekly.20220306_02:03:01
	item 40 key (256 DIR_ITEM 1705340674) itemoff 13996 itemsize 59
		location key (165305 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601957 data_len 0 name_len 29
		name: Sound_daily.20220327_01:02:01
	item 41 key (256 DIR_ITEM 1731806220) itemoff 13940 itemsize 56
		location key (165298 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601944 data_len 0 name_len 26
		name: Video_ro.20220327_00:35:50
	item 42 key (256 DIR_ITEM 1782151770) itemoff 13880 itemsize 60
		location key (165391 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602067 data_len 0 name_len 30
		name: other_hourly.20220327_23:01:01
	item 43 key (256 DIR_ITEM 1808375180) itemoff 13824 itemsize 56
		location key (163301 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597239 data_len 0 name_len 26
		name: Sound_ro.20220306_00:45:46
	item 44 key (256 DIR_ITEM 1925286825) itemoff 13768 itemsize 56
		location key (164621 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601060 data_len 0 name_len 26
		name: other_ro.20220320_00:25:35
	item 45 key (256 DIR_ITEM 2030955372) itemoff 13712 itemsize 56
		location key (163919 ROOT_ITEM 18446744073709551615) type DIR
		transid 1599632 data_len 0 name_len 26
		name: Sound_ro.20220313_00:37:25
	item 46 key (256 DIR_ITEM 2112197083) itemoff 13652 itemsize 60
		location key (165395 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602073 data_len 0 name_len 30
		name: other_hourly.20220328_00:01:01
	item 47 key (256 DIR_ITEM 2127973047) itemoff 13617 itemsize 35
		location key (11221 ROOT_ITEM 18446744073709551615) type DIR
		transid 4042 data_len 0 name_len 5
		name: other
	item 48 key (256 DIR_ITEM 2146234148) itemoff 13583 itemsize 34
		location key (11223 ROOT_ITEM 18446744073709551615) type DIR
		transid 4061 data_len 0 name_len 4
		name: Soft
	item 49 key (256 DIR_ITEM 2153154160) itemoff 13523 itemsize 60
		location key (165389 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602065 data_len 0 name_len 30
		name: Sound_hourly.20220327_23:01:01
	item 50 key (256 DIR_ITEM 2201886496) itemoff 13464 itemsize 59
		location key (165101 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601683 data_len 0 name_len 29
		name: other_daily.20220325_01:02:01
	item 51 key (256 DIR_ITEM 2279840979) itemoff 13408 itemsize 56
		location key (162629 ROOT_ITEM 18446744073709551615) type DIR
		transid 1596346 data_len 0 name_len 26
		name: other_ro.20220227_00:25:38
	item 52 key (256 DIR_ITEM 2313060016) itemoff 13348 itemsize 60
		location key (160496 ROOT_ITEM 18446744073709551615) type DIR
		transid 1592684 data_len 0 name_len 30
		name: media_weekly.20220206_02:03:01
	item 53 key (256 DIR_ITEM 2354277240) itemoff 13308 itemsize 40
		location key (1492 INODE_ITEM 0) type SYMLINK
		transid 1601938 data_len 0 name_len 10
		name: other_last
	item 54 key (256 DIR_ITEM 2355411712) itemoff 13248 itemsize 60
		location key (160494 ROOT_ITEM 18446744073709551615) type DIR
		transid 1592682 data_len 0 name_len 30
		name: Video_weekly.20220206_02:03:01
	item 55 key (256 DIR_ITEM 2377365299) itemoff 13192 itemsize 56
		location key (165297 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601941 data_len 0 name_len 26
		name: Sound_ro.20220327_00:35:22
	item 56 key (256 DIR_ITEM 2396976783) itemoff 13132 itemsize 60
		location key (163319 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597266 data_len 0 name_len 30
		name: other_weekly.20220306_02:03:01
	item 57 key (256 DIR_ITEM 2456187574) itemoff 13076 itemsize 56
		location key (163298 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597234 data_len 0 name_len 26
		name: media_ro.20220306_00:24:43
	item 58 key (256 DIR_ITEM 2467975514) itemoff 13021 itemsize 55
		location key (163300 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597237 data_len 0 name_len 25
		name: Soft_ro.20220306_00:33:55
	item 59 key (256 DIR_ITEM 2542431729) itemoff 12961 itemsize 60
		location key (165393 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602071 data_len 0 name_len 30
		name: Sound_hourly.20220328_00:01:01
	item 60 key (256 DIR_ITEM 2610074692) itemoff 12901 itemsize 60
		location key (165406 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602087 data_len 0 name_len 30
		name: media_hourly.20220328_02:01:01
	item 61 key (256 DIR_ITEM 2650936973) itemoff 12842 itemsize 59
		location key (165306 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601958 data_len 0 name_len 29
		name: media_daily.20220327_01:02:01
	item 62 key (256 DIR_ITEM 2653442548) itemoff 12782 itemsize 60
		location key (165404 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602085 data_len 0 name_len 30
		name: Video_hourly.20220328_02:01:01
	item 63 key (256 DIR_ITEM 2717852207) itemoff 12728 itemsize 54
		location key (163921 ROOT_ITEM 18446744073709551615) type DIR
		transid 1599637 data_len 0 name_len 24
		name: Win_ro.20220313_00:38:28
	item 64 key (256 DIR_ITEM 2745397786) itemoff 12669 itemsize 59
		location key (165402 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602082 data_len 0 name_len 29
		name: media_daily.20220328_01:02:01
	item 65 key (256 DIR_ITEM 2782562819) itemoff 12610 itemsize 59
		location key (165098 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601680 data_len 0 name_len 29
		name: Video_daily.20220325_01:02:01
	item 66 key (256 DIR_ITEM 2793356223) itemoff 12554 itemsize 56
		location key (163299 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597235 data_len 0 name_len 26
		name: other_ro.20220306_00:25:40
	item 67 key (256 DIR_ITEM 2811536125) itemoff 12519 itemsize 35
		location key (11288 ROOT_ITEM 18446744073709551615) type DIR
		transid 4174 data_len 0 name_len 5
		name: Video
	item 68 key (256 DIR_ITEM 2827986674) itemoff 12485 itemsize 34
		location key (1010 INODE_ITEM 0) type FILE
		transid 1455723 data_len 0 name_len 4
		name: copy
	item 69 key (256 DIR_ITEM 2917386315) itemoff 12425 itemsize 60
		location key (165407 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602088 data_len 0 name_len 30
		name: other_hourly.20220328_02:01:01
	item 70 key (256 DIR_ITEM 2935166119) itemoff 12366 itemsize 59
		location key (165199 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601810 data_len 0 name_len 29
		name: Sound_daily.20220326_01:02:01
	item 71 key (256 DIR_ITEM 2935255735) itemoff 12311 itemsize 55
		location key (165296 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601938 data_len 0 name_len 25
		name: Soft_ro.20220327_00:32:46
	item 72 key (256 DIR_ITEM 3098520192) itemoff 12251 itemsize 60
		location key (163318 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597265 data_len 0 name_len 30
		name: media_weekly.20220306_02:03:01
	item 73 key (256 DIR_ITEM 3170169652) itemoff 12195 itemsize 56
		location key (164624 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601068 data_len 0 name_len 26
		name: Video_ro.20220320_00:40:27
	item 74 key (256 DIR_ITEM 3172298544) itemoff 12135 itemsize 60
		location key (163316 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597263 data_len 0 name_len 30
		name: Video_weekly.20220306_02:03:01
	item 75 key (256 DIR_ITEM 3215925951) itemoff 12075 itemsize 60
		location key (160497 ROOT_ITEM 18446744073709551615) type DIR
		transid 1592685 data_len 0 name_len 30
		name: other_weekly.20220206_02:03:01
	item 76 key (256 DIR_ITEM 3236289372) itemoff 12015 itemsize 60
		location key (161200 ROOT_ITEM 18446744073709551615) type DIR
		transid 1594246 data_len 0 name_len 30
		name: other_weekly.20220213_02:03:01
	item 77 key (256 DIR_ITEM 3290748772) itemoff 11959 itemsize 56
		location key (163302 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597242 data_len 0 name_len 26
		name: Video_ro.20220306_00:46:14
	item 78 key (256 DIR_ITEM 3360445642) itemoff 11903 itemsize 56
		location key (162631 ROOT_ITEM 18446744073709551615) type DIR
		transid 1596351 data_len 0 name_len 26
		name: Sound_ro.20220227_00:40:41
	item 79 key (256 DIR_ITEM 3470690534) itemoff 11847 itemsize 56
		location key (164620 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601059 data_len 0 name_len 26
		name: media_ro.20220320_00:24:47
	item 80 key (256 DIR_ITEM 3550830046) itemoff 11787 itemsize 60
		location key (159785 ROOT_ITEM 18446744073709551615) type DIR
		transid 1591389 data_len 0 name_len 30
		name: Video_weekly.20220130_02:03:01
	item 81 key (256 DIR_ITEM 3592394862) itemoff 11727 itemsize 60
		location key (159787 ROOT_ITEM 18446744073709551615) type DIR
		transid 1591391 data_len 0 name_len 30
		name: media_weekly.20220130_02:03:01
	item 82 key (256 DIR_ITEM 3676890430) itemoff 11668 itemsize 59
		location key (165201 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601812 data_len 0 name_len 29
		name: other_daily.20220326_01:02:01
	item 83 key (256 DIR_ITEM 3765355617) itemoff 11608 itemsize 60
		location key (159788 ROOT_ITEM 18446744073709551615) type DIR
		transid 1591392 data_len 0 name_len 30
		name: other_weekly.20220130_02:03:01
	item 84 key (256 DIR_ITEM 3787964139) itemoff 11554 itemsize 54
		location key (163303 ROOT_ITEM 18446744073709551615) type DIR
		transid 1597244 data_len 0 name_len 24
		name: Win_ro.20220306_00:47:32
	item 85 key (256 DIR_ITEM 3799902853) itemoff 11514 itemsize 40
		location key (1491 INODE_ITEM 0) type SYMLINK
		transid 1601935 data_len 0 name_len 10
		name: media_last
	item 86 key (256 DIR_ITEM 3872736471) itemoff 11460 itemsize 54
		location key (162645 ROOT_ITEM 18446744073709551615) type DIR
		transid 1596372 data_len 0 name_len 24
		name: Win_ro.20220227_02:10:25
	item 87 key (256 DIR_ITEM 3951915213) itemoff 11420 itemsize 40
		location key (1495 INODE_ITEM 0) type SYMLINK
		transid 1601947 data_len 0 name_len 10
		name: Video_last
	item 88 key (256 DIR_ITEM 4079895267) itemoff 11360 itemsize 60
		location key (161197 ROOT_ITEM 18446744073709551615) type DIR
		transid 1594243 data_len 0 name_len 30
		name: Video_weekly.20220213_02:03:01
	item 89 key (256 DIR_ITEM 4084751325) itemoff 11321 itemsize 39
		location key (1493 INODE_ITEM 0) type SYMLINK
		transid 1601941 data_len 0 name_len 9
		name: Soft_last
	item 90 key (256 DIR_ITEM 4136929107) itemoff 11261 itemsize 60
		location key (161199 ROOT_ITEM 18446744073709551615) type DIR
		transid 1594245 data_len 0 name_len 30
		name: media_weekly.20220213_02:03:01
	item 91 key (256 DIR_ITEM 4142263993) itemoff 11202 itemsize 59
		location key (165099 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601681 data_len 0 name_len 29
		name: Sound_daily.20220325_01:02:01
	item 92 key (256 DIR_ITEM 4156034002) itemoff 11147 itemsize 55
		location key (162630 ROOT_ITEM 18446744073709551615) type DIR
		transid 1596348 data_len 0 name_len 25
		name: Soft_ro.20220227_00:34:48
	item 93 key (256 DIR_ITEM 4198095398) itemoff 11091 itemsize 56
		location key (165294 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601934 data_len 0 name_len 26
		name: media_ro.20220327_00:24:32
	item 94 key (256 DIR_ITEM 4258033693) itemoff 11032 itemsize 59
		location key (165198 ROOT_ITEM 18446744073709551615) type DIR
		transid 1601809 data_len 0 name_len 29
		name: Video_daily.20220326_01:02:01
	item 95 key (256 DIR_ITEM 4287185209) itemoff 10972 itemsize 60
		location key (165397 ROOT_ITEM 18446744073709551615) type DIR
		transid 1602076 data_len 0 name_len 30
		name: Sound_hourly.20220328_01:01:01
	item 96 key (256 DIR_INDEX 8) itemoff 10937 itemsize 35
		location key (11221 ROOT_ITEM 18446744073709551615) type DIR
		transid 4042 data_len 0 name_len 5
		name: other
	item 97 key (256 DIR_INDEX 9) itemoff 10902 itemsize 35
		location key (11222 ROOT_ITEM 18446744073709551615) type DIR
		transid 4058 data_len 0 name_len 5
		name: media
	item 98 key (256 DIR_INDEX 10) itemoff 10868 itemsize 34
		location key (11223 ROOT_ITEM 18446744073709551615) type DIR
		transid 4061 data_len 0 name_len 4
		name: Soft
	item 99 key (256 DIR_INDEX 11) itemoff 10835 itemsize 33
		location key (11224 ROOT_ITEM 18446744073709551615) type DIR
		transid 4062 data_len 0 name_len 3
		name: Win
	item 100 key (256 DIR_INDEX 17) itemoff 10800 itemsize 35
		location key (11288 ROOT_ITEM 18446744073709551615) type DIR
		transid 4174 data_len 0 name_len 5
		name: Video
	item 101 key (256 DIR_INDEX 51) itemoff 10765 itemsize 35
		location key (11322 ROOT_ITEM 18446744073709551615) type DIR
		transid 4223 data_len 0 name_len 5
		name: Sound
1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m Kexit

Script done on Mon 04 Apr 2022 18:38:35 PDT
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  1:42                                                                                               ` Marc MERLIN
@ 2022-04-05  1:55                                                                                                 ` Josef Bacik
  2022-04-05  2:07                                                                                                   ` Marc MERLIN
  2022-04-05 16:22                                                                                                 ` Roman Mamedov
  2022-04-05 18:38                                                                                                 ` Zygo Blaxell
  2 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05  1:55 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Apr 4, 2022 at 9:43 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Apr 04, 2022 at 09:22:26PM -0400, Josef Bacik wrote:
> > Ok lets try again with -b 13577821667328, and if the owner doesn't say
> > ROOT_TREE try 13577775284224 and then 13577814573056.  Thanks,
>
> EXTENT_TREE, FS_TREE, and FS_TREE
>
> And shit, I got distracted and sent the text output to
> /dev/mapper/dshelf1a, so I clobbered about 30K of the device.
> I'm assuming there was probably something there?

Yes the chunk tree, but your FS is DUP so we'll find the other copy,
it'll be fine.

Do you have any subvolumes that aren't snapshots?  I think I can
re-create everything without knowing what other roots there are, but
if you have snapshots/subvolumes I need to be more careful.

In any case I'm going to go to bed, it's been a long week.  I'll start
working on a tool to re-create your root tree in the morning, and then
hopefully we can go from there.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  1:55                                                                                                 ` Josef Bacik
@ 2022-04-05  2:07                                                                                                   ` Marc MERLIN
  2022-04-05 14:11                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05  2:07 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Apr 04, 2022 at 09:55:41PM -0400, Josef Bacik wrote:
> On Mon, Apr 4, 2022 at 9:43 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Mon, Apr 04, 2022 at 09:22:26PM -0400, Josef Bacik wrote:
> > > Ok lets try again with -b 13577821667328, and if the owner doesn't say
> > > ROOT_TREE try 13577775284224 and then 13577814573056.  Thanks,
> >
> > EXTENT_TREE, FS_TREE, and FS_TREE
> >
> > And shit, I got distracted and sent the text output to
> > /dev/mapper/dshelf1a, so I clobbered about 30K of the device.
> > I'm assuming there was probably something there?
> 
> Yes the chunk tree, but your FS is DUP so we'll find the other copy,
> it'll be fine.

Great news, thanks.

> Do you have any subvolumes that aren't snapshots?  I think I can
> re-create everything without knowing what other roots there are, but
> if you have snapshots/subvolumes I need to be more careful.
 
6 subvolumes plus snwpshots of them:
media other Soft Sound Video Win

Maybe 50 to 100 snapshots total (mostly snapshots)

> In any case I'm going to go to bed, it's been a long week.  I'll start
> working on a tool to re-create your root tree in the morning, and then
> hopefully we can go from there.  Thanks,

Thanks for your help, have a good night.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  2:07                                                                                                   ` Marc MERLIN
@ 2022-04-05 14:11                                                                                                     ` Josef Bacik
  2022-04-05 15:53                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05 14:11 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Apr 4, 2022 at 10:07 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Apr 04, 2022 at 09:55:41PM -0400, Josef Bacik wrote:
> > On Mon, Apr 4, 2022 at 9:43 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Mon, Apr 04, 2022 at 09:22:26PM -0400, Josef Bacik wrote:
> > > > Ok lets try again with -b 13577821667328, and if the owner doesn't say
> > > > ROOT_TREE try 13577775284224 and then 13577814573056.  Thanks,
> > >
> > > EXTENT_TREE, FS_TREE, and FS_TREE
> > >
> > > And shit, I got distracted and sent the text output to
> > > /dev/mapper/dshelf1a, so I clobbered about 30K of the device.
> > > I'm assuming there was probably something there?
> >
> > Yes the chunk tree, but your FS is DUP so we'll find the other copy,
> > it'll be fine.
>
> Great news, thanks.
>
> > Do you have any subvolumes that aren't snapshots?  I think I can
> > re-create everything without knowing what other roots there are, but
> > if you have snapshots/subvolumes I need to be more careful.
>
> 6 subvolumes plus snwpshots of them:
> media other Soft Sound Video Win
>
> Maybe 50 to 100 snapshots total (mostly snapshots)
>
> > In any case I'm going to go to bed, it's been a long week.  I'll start
> > working on a tool to re-create your root tree in the morning, and then
> > hopefully we can go from there.  Thanks,
>
> Thanks for your help, have a good night.
>

Ok the fs volumes are going to be trickier to put back together, so
first lets make sure the fs tree pointer is correct and relatively
intact.  Can you do

./btrfs inspect-internal dump-tree -b 13577799401472 /dev/whatever

I don't need the full dump since it's all the metadata for your fs
tree.  What I'm looking for is if this looks sane, you'll see the
normal errors you see from the fs being generally fucked, but I want
this to spew and seem like it worked, and all have owner of FS_ROOT.

The problem here is that since the tree root is fucked I can't easily
find the subvolumes bytenr, so we're going to have to walk the FS_ROOT
and find the ID's for all your subvolumes, and then use
btrfs-find-root to find the bytenrs of those roots.  From there I can
clear your old tree root and create it with just the subvolumes, chunk
root, and device root.  From there fsck can re-create the extent and
csum tree.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 14:11                                                                                                     ` Josef Bacik
@ 2022-04-05 15:53                                                                                                       ` Marc MERLIN
  2022-04-05 15:55                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05 15:53 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 10:11:25AM -0400, Josef Bacik wrote:
> Ok the fs volumes are going to be trickier to put back together, so
> first lets make sure the fs tree pointer is correct and relatively
> intact.  Can you do
> 
> ./btrfs inspect-internal dump-tree -b 13577799401472 /dev/whatever
> 
> I don't need the full dump since it's all the metadata for your fs
> tree.  What I'm looking for is if this looks sane, you'll see the
> normal errors you see from the fs being generally fucked, but I want
> this to spew and seem like it worked, and all have owner of FS_ROOT.

Thanks. It's not too big, attached


gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 13577799401472 /dev/mapper/dshelf1a &>/tmp/d
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
FS_INFO IS 0x55741d8f7470
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x55741d8f7470
btrfs-progs v5.16.2 
leaf 13577799401472 items 234 free space 2612 generation 1602242 owner EXTENT_TREE
leaf 13577799401472 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
	item 0 key (1901621264384 METADATA_ITEM 0) itemoff 16250 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 1 key (1901621280768 METADATA_ITEM 0) itemoff 16217 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 2 key (1901621297152 METADATA_ITEM 0) itemoff 16184 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 3 key (1901621313536 METADATA_ITEM 0) itemoff 16151 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 4 key (1901621329920 METADATA_ITEM 0) itemoff 16118 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 5 key (1901621346304 METADATA_ITEM 0) itemoff 16085 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 6 key (1901621362688 METADATA_ITEM 0) itemoff 16052 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 7 key (1901621379072 METADATA_ITEM 0) itemoff 16019 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 8 key (1901621395456 METADATA_ITEM 0) itemoff 15986 itemsize 33
		refs 1 gen 2327 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 9 key (1901621411840 METADATA_ITEM 0) itemoff 15953 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 10 key (1901621428224 METADATA_ITEM 0) itemoff 15920 itemsize 33
		refs 1 gen 2327 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 11 key (1901621444608 METADATA_ITEM 0) itemoff 15887 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 12 key (1901621460992 METADATA_ITEM 0) itemoff 15854 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 13 key (1901621477376 METADATA_ITEM 0) itemoff 15821 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 14 key (1901621493760 METADATA_ITEM 0) itemoff 15788 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 15 key (1901621510144 METADATA_ITEM 0) itemoff 15755 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 16 key (1901621526528 METADATA_ITEM 0) itemoff 15722 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 17 key (1901621542912 METADATA_ITEM 0) itemoff 15689 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 18 key (1901621559296 METADATA_ITEM 0) itemoff 15656 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 19 key (1901621575680 METADATA_ITEM 0) itemoff 15623 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 20 key (1901621592064 METADATA_ITEM 0) itemoff 15590 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 21 key (1901621608448 METADATA_ITEM 0) itemoff 15557 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 22 key (1901621624832 METADATA_ITEM 0) itemoff 15524 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 23 key (1901621641216 METADATA_ITEM 0) itemoff 15491 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 24 key (1901621657600 METADATA_ITEM 0) itemoff 15458 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 25 key (1901621673984 METADATA_ITEM 0) itemoff 15425 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 26 key (1901621690368 METADATA_ITEM 0) itemoff 15392 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 27 key (1901621706752 METADATA_ITEM 0) itemoff 15359 itemsize 33
		refs 1 gen 2955 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 28 key (1901621723136 METADATA_ITEM 0) itemoff 15326 itemsize 33
		refs 1 gen 2955 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 29 key (1901621755904 METADATA_ITEM 0) itemoff 15293 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 30 key (1901621772288 METADATA_ITEM 0) itemoff 15260 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 31 key (1901621788672 METADATA_ITEM 0) itemoff 15227 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 32 key (1901621805056 METADATA_ITEM 0) itemoff 15194 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 33 key (1901621821440 METADATA_ITEM 0) itemoff 15161 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 34 key (1901621837824 METADATA_ITEM 0) itemoff 15128 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 35 key (1901621854208 METADATA_ITEM 0) itemoff 15095 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 36 key (1901621870592 METADATA_ITEM 0) itemoff 15062 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 37 key (1901621886976 METADATA_ITEM 0) itemoff 15029 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 38 key (1901621903360 METADATA_ITEM 0) itemoff 14996 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 39 key (1901621919744 METADATA_ITEM 0) itemoff 14963 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 40 key (1901621936128 METADATA_ITEM 0) itemoff 14930 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 41 key (1901621952512 METADATA_ITEM 0) itemoff 14897 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 42 key (1901621968896 METADATA_ITEM 0) itemoff 14864 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 43 key (1901621985280 METADATA_ITEM 0) itemoff 14831 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 44 key (1901622001664 METADATA_ITEM 0) itemoff 14798 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 45 key (1901622018048 METADATA_ITEM 0) itemoff 14765 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 46 key (1901622034432 METADATA_ITEM 0) itemoff 14732 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 47 key (1901622050816 METADATA_ITEM 0) itemoff 14699 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 48 key (1901622067200 METADATA_ITEM 0) itemoff 14666 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 49 key (1901622083584 METADATA_ITEM 0) itemoff 14633 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 50 key (1901622099968 METADATA_ITEM 0) itemoff 14600 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 51 key (1901622116352 METADATA_ITEM 0) itemoff 14567 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 52 key (1901622132736 METADATA_ITEM 0) itemoff 14534 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 53 key (1901622149120 METADATA_ITEM 0) itemoff 14501 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 54 key (1901622165504 METADATA_ITEM 0) itemoff 14468 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 55 key (1901622181888 METADATA_ITEM 0) itemoff 14435 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 56 key (1901622198272 METADATA_ITEM 0) itemoff 14402 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 57 key (1901622214656 METADATA_ITEM 0) itemoff 14369 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 58 key (1901622231040 METADATA_ITEM 0) itemoff 14336 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 59 key (1901622247424 METADATA_ITEM 0) itemoff 14303 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 60 key (1901622263808 METADATA_ITEM 0) itemoff 14270 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 61 key (1901622280192 METADATA_ITEM 0) itemoff 14237 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 62 key (1901622296576 METADATA_ITEM 0) itemoff 14204 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 63 key (1901622312960 METADATA_ITEM 0) itemoff 14171 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 64 key (1901622329344 METADATA_ITEM 0) itemoff 14138 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 65 key (1901622345728 METADATA_ITEM 0) itemoff 14105 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 66 key (1901622362112 METADATA_ITEM 0) itemoff 14072 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 67 key (1901622378496 METADATA_ITEM 0) itemoff 14039 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 68 key (1901622394880 METADATA_ITEM 0) itemoff 14006 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 69 key (1901622411264 METADATA_ITEM 0) itemoff 13973 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 70 key (1901622427648 METADATA_ITEM 0) itemoff 13940 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 71 key (1901622444032 METADATA_ITEM 0) itemoff 13907 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 72 key (1901622460416 METADATA_ITEM 0) itemoff 13874 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 73 key (1901622476800 METADATA_ITEM 0) itemoff 13841 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 74 key (1901622493184 METADATA_ITEM 0) itemoff 13808 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 75 key (1901622509568 METADATA_ITEM 0) itemoff 13775 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 76 key (1901622542336 METADATA_ITEM 0) itemoff 13742 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 77 key (1901622558720 METADATA_ITEM 0) itemoff 13709 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 78 key (1901622575104 METADATA_ITEM 0) itemoff 13676 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 79 key (1901622591488 METADATA_ITEM 0) itemoff 13643 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 80 key (1901622607872 METADATA_ITEM 0) itemoff 13610 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 81 key (1901622624256 METADATA_ITEM 0) itemoff 13577 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 82 key (1901622640640 METADATA_ITEM 0) itemoff 13544 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 83 key (1901622657024 METADATA_ITEM 0) itemoff 13511 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 84 key (1901622673408 METADATA_ITEM 0) itemoff 13478 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 85 key (1901622689792 METADATA_ITEM 0) itemoff 13445 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 86 key (1901622706176 METADATA_ITEM 0) itemoff 13412 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 87 key (1901622722560 METADATA_ITEM 0) itemoff 13379 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 88 key (1901622738944 METADATA_ITEM 0) itemoff 13346 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 89 key (1901622755328 METADATA_ITEM 0) itemoff 13313 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 90 key (1901622771712 METADATA_ITEM 0) itemoff 13280 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 91 key (1901622788096 METADATA_ITEM 0) itemoff 13247 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 92 key (1901622804480 METADATA_ITEM 0) itemoff 13214 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 93 key (1901622820864 METADATA_ITEM 0) itemoff 13181 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 94 key (1901622837248 METADATA_ITEM 0) itemoff 13148 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 95 key (1901622853632 METADATA_ITEM 0) itemoff 13115 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 96 key (1901622870016 METADATA_ITEM 0) itemoff 13082 itemsize 33
		refs 1 gen 2327 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 97 key (1901622886400 METADATA_ITEM 0) itemoff 13049 itemsize 33
		refs 1 gen 2327 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 98 key (1901622902784 METADATA_ITEM 0) itemoff 13016 itemsize 33
		refs 1 gen 2327 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 99 key (1901622919168 METADATA_ITEM 0) itemoff 12983 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 100 key (1901622935552 METADATA_ITEM 0) itemoff 12950 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 101 key (1901622951936 METADATA_ITEM 0) itemoff 12917 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 102 key (1901622968320 METADATA_ITEM 0) itemoff 12884 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 103 key (1901622984704 METADATA_ITEM 0) itemoff 12851 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 104 key (1901623001088 METADATA_ITEM 0) itemoff 12818 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 105 key (1901623017472 METADATA_ITEM 0) itemoff 12785 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 106 key (1901623033856 METADATA_ITEM 0) itemoff 12752 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 107 key (1901623050240 METADATA_ITEM 0) itemoff 12719 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 108 key (1901623066624 METADATA_ITEM 0) itemoff 12686 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 109 key (1901623083008 METADATA_ITEM 0) itemoff 12653 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 110 key (1901623099392 METADATA_ITEM 0) itemoff 12620 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 111 key (1901623115776 METADATA_ITEM 0) itemoff 12587 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 112 key (1901623132160 METADATA_ITEM 0) itemoff 12554 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 113 key (1901623148544 METADATA_ITEM 0) itemoff 12521 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 114 key (1901623164928 METADATA_ITEM 0) itemoff 12488 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 115 key (1901623181312 METADATA_ITEM 0) itemoff 12455 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 116 key (1901623197696 METADATA_ITEM 0) itemoff 12422 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 117 key (1901623214080 METADATA_ITEM 0) itemoff 12389 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 118 key (1901623230464 METADATA_ITEM 0) itemoff 12356 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 119 key (1901623246848 METADATA_ITEM 0) itemoff 12323 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 120 key (1901623263232 METADATA_ITEM 0) itemoff 12290 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 121 key (1901623279616 METADATA_ITEM 0) itemoff 12257 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 122 key (1901623296000 METADATA_ITEM 0) itemoff 12224 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 123 key (1901623312384 METADATA_ITEM 0) itemoff 12191 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 124 key (1901623328768 METADATA_ITEM 0) itemoff 12158 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 125 key (1901623345152 METADATA_ITEM 0) itemoff 12125 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 126 key (1901623361536 METADATA_ITEM 0) itemoff 12092 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 127 key (1901623377920 METADATA_ITEM 0) itemoff 12059 itemsize 33
		refs 1 gen 1526014 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 128 key (1901623394304 METADATA_ITEM 0) itemoff 12026 itemsize 33
		refs 1 gen 1602167 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 129 key (1901623410688 METADATA_ITEM 0) itemoff 11993 itemsize 33
		refs 1 gen 2384 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 130 key (1901623427072 METADATA_ITEM 0) itemoff 11960 itemsize 33
		refs 1 gen 1482589 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 131 key (1901623476224 METADATA_ITEM 0) itemoff 11927 itemsize 33
		refs 1 gen 1526014 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 132 key (1901623492608 METADATA_ITEM 0) itemoff 11894 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 133 key (1901623508992 METADATA_ITEM 0) itemoff 11861 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 134 key (1901623525376 METADATA_ITEM 0) itemoff 11828 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 135 key (1901623541760 METADATA_ITEM 0) itemoff 11795 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 136 key (1901623558144 METADATA_ITEM 0) itemoff 11762 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 137 key (1901623574528 METADATA_ITEM 0) itemoff 11729 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 138 key (1901623590912 METADATA_ITEM 0) itemoff 11696 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 139 key (1901623607296 METADATA_ITEM 0) itemoff 11663 itemsize 33
		refs 1 gen 1482589 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 140 key (1901623623680 METADATA_ITEM 0) itemoff 11630 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 141 key (1901623640064 METADATA_ITEM 0) itemoff 11597 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 142 key (1901623656448 METADATA_ITEM 0) itemoff 11564 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 143 key (1901623672832 METADATA_ITEM 0) itemoff 11531 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 144 key (1901623689216 METADATA_ITEM 0) itemoff 11498 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 145 key (1901623705600 METADATA_ITEM 0) itemoff 11465 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 146 key (1901623721984 METADATA_ITEM 0) itemoff 11432 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 147 key (1901623738368 METADATA_ITEM 0) itemoff 11399 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 148 key (1901623754752 METADATA_ITEM 0) itemoff 11366 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 149 key (1901623771136 METADATA_ITEM 0) itemoff 11333 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 150 key (1901623787520 METADATA_ITEM 0) itemoff 11300 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 151 key (1901623803904 METADATA_ITEM 0) itemoff 11267 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 152 key (1901623820288 METADATA_ITEM 0) itemoff 11234 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 153 key (1901623836672 METADATA_ITEM 0) itemoff 11201 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 154 key (1901623853056 METADATA_ITEM 0) itemoff 11168 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 155 key (1901623869440 METADATA_ITEM 0) itemoff 11135 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 156 key (1901623885824 METADATA_ITEM 0) itemoff 11102 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 157 key (1901623902208 METADATA_ITEM 0) itemoff 11069 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 158 key (1901623918592 METADATA_ITEM 0) itemoff 11036 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 159 key (1901623934976 METADATA_ITEM 0) itemoff 11003 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 160 key (1901623951360 METADATA_ITEM 0) itemoff 10970 itemsize 33
		refs 1 gen 1125 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 161 key (1901623967744 METADATA_ITEM 0) itemoff 10937 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 162 key (1901623984128 METADATA_ITEM 0) itemoff 10904 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 163 key (1901624000512 METADATA_ITEM 0) itemoff 10871 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 164 key (1901624016896 METADATA_ITEM 0) itemoff 10838 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 165 key (1901624033280 METADATA_ITEM 0) itemoff 10805 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 166 key (1901624049664 METADATA_ITEM 0) itemoff 10772 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 167 key (1901624066048 METADATA_ITEM 0) itemoff 10739 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 168 key (1901624082432 METADATA_ITEM 0) itemoff 10706 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 169 key (1901624098816 METADATA_ITEM 0) itemoff 10673 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 170 key (1901624115200 METADATA_ITEM 0) itemoff 10640 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 171 key (1901624131584 METADATA_ITEM 0) itemoff 10607 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 172 key (1901624147968 METADATA_ITEM 0) itemoff 10574 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 173 key (1901624164352 METADATA_ITEM 0) itemoff 10541 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 174 key (1901624180736 METADATA_ITEM 0) itemoff 10508 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 175 key (1901624197120 METADATA_ITEM 0) itemoff 10475 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 176 key (1901624213504 METADATA_ITEM 0) itemoff 10442 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 177 key (1901624229888 METADATA_ITEM 0) itemoff 10409 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 178 key (1901624246272 METADATA_ITEM 0) itemoff 10376 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 179 key (1901624262656 METADATA_ITEM 0) itemoff 10343 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 180 key (1901624279040 METADATA_ITEM 0) itemoff 10310 itemsize 33
		refs 1 gen 2384 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 181 key (1901624295424 METADATA_ITEM 0) itemoff 10277 itemsize 33
		refs 1 gen 2384 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 182 key (1901624311808 METADATA_ITEM 0) itemoff 10244 itemsize 33
		refs 1 gen 2384 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 183 key (1901624328192 METADATA_ITEM 0) itemoff 10211 itemsize 33
		refs 1 gen 2384 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 184 key (1901624344576 METADATA_ITEM 0) itemoff 10178 itemsize 33
		refs 1 gen 2384 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 185 key (1901624360960 METADATA_ITEM 0) itemoff 10145 itemsize 33
		refs 1 gen 2384 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 186 key (1901624377344 METADATA_ITEM 0) itemoff 10112 itemsize 33
		refs 1 gen 2384 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 187 key (1901624393728 METADATA_ITEM 0) itemoff 10079 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 188 key (1901624410112 METADATA_ITEM 0) itemoff 10046 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 189 key (1901624426496 METADATA_ITEM 0) itemoff 10013 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 190 key (1901624442880 METADATA_ITEM 0) itemoff 9980 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 191 key (1901624459264 METADATA_ITEM 0) itemoff 9947 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 192 key (1901624475648 METADATA_ITEM 0) itemoff 9914 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 193 key (1901624492032 METADATA_ITEM 0) itemoff 9881 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 194 key (1901624508416 METADATA_ITEM 0) itemoff 9848 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 195 key (1901624524800 METADATA_ITEM 0) itemoff 9815 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 196 key (1901624541184 METADATA_ITEM 0) itemoff 9782 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 197 key (1901624573952 METADATA_ITEM 0) itemoff 9749 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 198 key (1901624590336 METADATA_ITEM 0) itemoff 9716 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 199 key (1901624606720 METADATA_ITEM 0) itemoff 9683 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 200 key (1901624623104 METADATA_ITEM 0) itemoff 9650 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 201 key (1901624639488 METADATA_ITEM 0) itemoff 9617 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 202 key (1901624655872 METADATA_ITEM 0) itemoff 9584 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 203 key (1901624672256 METADATA_ITEM 0) itemoff 9551 itemsize 33
		refs 1 gen 3116 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 204 key (1901624688640 METADATA_ITEM 0) itemoff 9518 itemsize 33
		refs 1 gen 900861 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 205 key (1901624737792 METADATA_ITEM 0) itemoff 9485 itemsize 33
		refs 1 gen 1602167 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 206 key (1901624770560 METADATA_ITEM 0) itemoff 9452 itemsize 33
		refs 1 gen 1482589 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 207 key (1901624786944 METADATA_ITEM 0) itemoff 9419 itemsize 33
		refs 1 gen 1482589 flags TREE_BLOCK
		tree block skinny level 0
		shared block backref parent 15645446602752
	item 208 key (1901624819712 METADATA_ITEM 0) itemoff 9386 itemsize 33
		refs 1 gen 482803 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 209 key (1901624868864 METADATA_ITEM 0) itemoff 9353 itemsize 33
		refs 1 gen 1526014 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 210 key (1901624885248 METADATA_ITEM 0) itemoff 9320 itemsize 33
		refs 1 gen 1601407 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 211 key (1901624934400 METADATA_ITEM 0) itemoff 9287 itemsize 33
		refs 1 gen 1601407 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 212 key (1901624950784 METADATA_ITEM 0) itemoff 9254 itemsize 33
		refs 1 gen 1601407 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 213 key (1901624967168 METADATA_ITEM 0) itemoff 9221 itemsize 33
		refs 1 gen 12204 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 214 key (1901624983552 METADATA_ITEM 0) itemoff 9188 itemsize 33
		refs 1 gen 1482589 flags TREE_BLOCK
		tree block skinny level 0
		shared block backref parent 15645446602752
	item 215 key (1901625032704 METADATA_ITEM 0) itemoff 9155 itemsize 33
		refs 1 gen 300256 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root EXTENT_TREE
	item 216 key (1901625065472 METADATA_ITEM 0) itemoff 9122 itemsize 33
		refs 1 gen 1602167 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 217 key (1901625081856 METADATA_ITEM 0) itemoff 9089 itemsize 33
		refs 1 gen 1509218 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 218 key (1901625098240 METADATA_ITEM 0) itemoff 9056 itemsize 33
		refs 1 gen 4427 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 219 key (1901625114624 METADATA_ITEM 0) itemoff 9023 itemsize 33
		refs 1 gen 4427 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 220 key (1901625131008 METADATA_ITEM 0) itemoff 8990 itemsize 33
		refs 1 gen 4427 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 221 key (1901625147392 METADATA_ITEM 0) itemoff 8957 itemsize 33
		refs 1 gen 4427 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 222 key (1901625163776 METADATA_ITEM 0) itemoff 8924 itemsize 33
		refs 1 gen 1602167 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 223 key (1901625180160 METADATA_ITEM 0) itemoff 8891 itemsize 33
		refs 1 gen 1602167 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 224 key (1901625196544 METADATA_ITEM 0) itemoff 8858 itemsize 33
		refs 1 gen 3682 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 225 key (1901625212928 METADATA_ITEM 0) itemoff 8825 itemsize 33
		refs 1 gen 3682 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 226 key (1901625229312 METADATA_ITEM 0) itemoff 8792 itemsize 33
		refs 1 gen 2529 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 227 key (1901625245696 METADATA_ITEM 0) itemoff 8759 itemsize 33
		refs 1 gen 2529 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 228 key (1901625262080 METADATA_ITEM 0) itemoff 8726 itemsize 33
		refs 1 gen 2529 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 229 key (1901625278464 METADATA_ITEM 0) itemoff 8693 itemsize 33
		refs 1 gen 2529 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 230 key (1901625294848 METADATA_ITEM 0) itemoff 8660 itemsize 33
		refs 1 gen 2529 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 231 key (1901625311232 METADATA_ITEM 0) itemoff 8627 itemsize 33
		refs 1 gen 2529 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 232 key (1901625327616 METADATA_ITEM 0) itemoff 8594 itemsize 33
		refs 1 gen 2529 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root CSUM_TREE
	item 233 key (1901625344000 METADATA_ITEM 0) itemoff 8462 itemsize 132
		refs 12 gen 1526014 flags TREE_BLOCK
		tree block skinny level 0
		tree block backref root 11221
		shared block backref parent 15646057824256
		shared block backref parent 15645373939712
		shared block backref parent 15645039509504
		shared block backref parent 13577558376448
		shared block backref parent 12512208764928
		shared block backref parent 11971124183040
		shared block backref parent 11822367277056
		shared block backref parent 11822283620352
		shared block backref parent 11651814146048
		shared block backref parent 4866924920832
		shared block backref parent 782827061248
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 15:53                                                                                                       ` Marc MERLIN
@ 2022-04-05 15:55                                                                                                         ` Josef Bacik
  2022-04-05 17:41                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05 15:55 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 5, 2022 at 11:53 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Apr 05, 2022 at 10:11:25AM -0400, Josef Bacik wrote:
> > Ok the fs volumes are going to be trickier to put back together, so
> > first lets make sure the fs tree pointer is correct and relatively
> > intact.  Can you do
> >
> > ./btrfs inspect-internal dump-tree -b 13577799401472 /dev/whatever
> >
> > I don't need the full dump since it's all the metadata for your fs
> > tree.  What I'm looking for is if this looks sane, you'll see the
> > normal errors you see from the fs being generally fucked, but I want
> > this to spew and seem like it worked, and all have owner of FS_ROOT.
>
> Thanks. It's not too big, attached
>
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 13577799401472 /dev/mapper/dshelf1a &>/tmp/d
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> Ignoring transid failure
> FS_INFO IS 0x55741d8f7470
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Ignoring transid failure
> Couldn't find the last root for 4
> Couldn't setup device tree
> FS_INFO AFTER IS 0x55741d8f7470
> btrfs-progs v5.16.2
> leaf 13577799401472 items 234 free space 2612 generation 1602242 owner EXTENT_TREE

Ok this is what I was afraid of, but I anticipated this wrinkle and
I'm writing up something special to try and put everything back
together.  I'll have something wonky to try in a couple of hours.
Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  1:42                                                                                               ` Marc MERLIN
  2022-04-05  1:55                                                                                                 ` Josef Bacik
@ 2022-04-05 16:22                                                                                                 ` Roman Mamedov
  2022-04-05 22:06                                                                                                   ` Marc MERLIN
  2022-04-05 18:38                                                                                                 ` Zygo Blaxell
  2 siblings, 1 reply; 479+ messages in thread
From: Roman Mamedov @ 2022-04-05 16:22 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Josef Bacik, linux-btrfs

On Mon, 4 Apr 2022 18:42:59 -0700
Marc MERLIN <marc@merlins.org> wrote:

> And shit, I got distracted and sent the text output to
> /dev/mapper/dshelf1a, so I clobbered about 30K of the device. 
> I'm assuming there was probably something there?
> 
> Script started on Mon 04 Apr 2022 18:36:51 PDT
> 1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m ./btrfs-inspect-internal dump-tree -b 13577821667328 &>/dev/mapper/dshelf1a
> btrfs inspect-internal dump-tree: not enough arguments: 0 but at least 1 expected

It doesn't appear like it would write 30K to the device, it failed right away
since you didn't *specify* the device; the above output was to STDERR, and
there would be no writes to STDOUT (i.e. the device). Did you check the
content of /dev/mapper/dshelf1a (with less -f, for example)?

I know Josef said this is no prolem, but anyways. :)

-- 
With respect,
Roman

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 15:55                                                                                                         ` Josef Bacik
@ 2022-04-05 17:41                                                                                                           ` Josef Bacik
  2022-04-05 18:11                                                                                                             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05 17:41 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 5, 2022 at 11:55 AM Josef Bacik <josef@toxicpanda.com> wrote:
>
> On Tue, Apr 5, 2022 at 11:53 AM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Tue, Apr 05, 2022 at 10:11:25AM -0400, Josef Bacik wrote:
> > > Ok the fs volumes are going to be trickier to put back together, so
> > > first lets make sure the fs tree pointer is correct and relatively
> > > intact.  Can you do
> > >
> > > ./btrfs inspect-internal dump-tree -b 13577799401472 /dev/whatever
> > >
> > > I don't need the full dump since it's all the metadata for your fs
> > > tree.  What I'm looking for is if this looks sane, you'll see the
> > > normal errors you see from the fs being generally fucked, but I want
> > > this to spew and seem like it worked, and all have owner of FS_ROOT.
> >
> > Thanks. It's not too big, attached
> >
> >
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 13577799401472 /dev/mapper/dshelf1a &>/tmp/d
> > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > Ignoring transid failure
> > FS_INFO IS 0x55741d8f7470
> > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > Ignoring transid failure
> > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > Ignoring transid failure
> > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > Ignoring transid failure
> > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > Ignoring transid failure
> > Couldn't find the last root for 4
> > Couldn't setup device tree
> > FS_INFO AFTER IS 0x55741d8f7470
> > btrfs-progs v5.16.2
> > leaf 13577799401472 items 234 free space 2612 generation 1602242 owner EXTENT_TREE
>
> Ok this is what I was afraid of, but I anticipated this wrinkle and
> I'm writing up something special to try and put everything back
> together.  I'll have something wonky to try in a couple of hours.
> Thanks,
>

Ok I think I'm going to do a humpty-dumpty thing where I scan for the
best possible copies of blocks and stitch them into the tree root.
I've wired something up to find out how many bad slots we're working
with so I've got an idea of how careful I need to be, can you pull
your branch again, rebuild, and run

./btrfs-find-root -o 1 /dev/whatever

and send me the output?  I'm working on the tool right now, hoping it
takes me a couple of hours at most (mostly because I have meetings
sporadically throughout today).  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 17:41                                                                                                           ` Josef Bacik
@ 2022-04-05 18:11                                                                                                             ` Marc MERLIN
  2022-04-05 18:36                                                                                                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05 18:11 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 01:41:10PM -0400, Josef Bacik wrote:
> Ok I think I'm going to do a humpty-dumpty thing where I scan for the
> best possible copies of blocks and stitch them into the tree root.
> I've wired something up to find out how many bad slots we're working
> with so I've got an idea of how careful I need to be, can you pull
> your branch again, rebuild, and run
> 
> ./btrfs-find-root -o 1 /dev/whatever
> 
> and send me the output?  I'm working on the tool right now, hoping it
> takes me a couple of hours at most (mostly because I have meetings
> sporadically throughout today).  Thanks,

Cool. there you go

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1
/dev/mapper/dshelf1a
Couldn't read chunk tree
WTF???
ERROR: open ctree failed

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 18:11                                                                                                             ` Marc MERLIN
@ 2022-04-05 18:36                                                                                                               ` Josef Bacik
  2022-04-05 19:51                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05 18:36 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 5, 2022 at 2:11 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Apr 05, 2022 at 01:41:10PM -0400, Josef Bacik wrote:
> > Ok I think I'm going to do a humpty-dumpty thing where I scan for the
> > best possible copies of blocks and stitch them into the tree root.
> > I've wired something up to find out how many bad slots we're working
> > with so I've got an idea of how careful I need to be, can you pull
> > your branch again, rebuild, and run
> >
> > ./btrfs-find-root -o 1 /dev/whatever
> >
> > and send me the output?  I'm working on the tool right now, hoping it
> > takes me a couple of hours at most (mostly because I have meetings
> > sporadically throughout today).  Thanks,
>
> Cool. there you go
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1
> /dev/mapper/dshelf1a
> Couldn't read chunk tree
> WTF???
> ERROR: open ctree failed
>

That's new, the chunk tree wasn't failing before right?  Anyway I
pushed a change, it should work now, thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05  1:42                                                                                               ` Marc MERLIN
  2022-04-05  1:55                                                                                                 ` Josef Bacik
  2022-04-05 16:22                                                                                                 ` Roman Mamedov
@ 2022-04-05 18:38                                                                                                 ` Zygo Blaxell
  2022-04-05 19:31                                                                                                   ` Marc MERLIN
  2 siblings, 1 reply; 479+ messages in thread
From: Zygo Blaxell @ 2022-04-05 18:38 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Josef Bacik, linux-btrfs

On Mon, Apr 04, 2022 at 06:42:59PM -0700, Marc MERLIN wrote:
> On Mon, Apr 04, 2022 at 09:22:26PM -0400, Josef Bacik wrote:
> > Ok lets try again with -b 13577821667328, and if the owner doesn't say
> > ROOT_TREE try 13577775284224 and then 13577814573056.  Thanks,
> 
> EXTENT_TREE, FS_TREE, and FS_TREE
> 
> And shit, I got distracted and sent the text output to
> /dev/mapper/dshelf1a, so I clobbered about 30K of the device. 
> I'm assuming there was probably something there?

First superblock is at 64K, and the first 1M of every device is unused.
A 30K write at the beginning will overwrite nothing important to btrfs.

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 18:38                                                                                                 ` Zygo Blaxell
@ 2022-04-05 19:31                                                                                                   ` Marc MERLIN
  0 siblings, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05 19:31 UTC (permalink / raw)
  To: Zygo Blaxell, Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 02:38:59PM -0400, Zygo Blaxell wrote:
> First superblock is at 64K, and the first 1M of every device is unused.
> A 30K write at the beginning will overwrite nothing important to btrfs.

I figured something like that was the case, thanks for confirming.

On Tue, Apr 05, 2022 at 02:36:29PM -0400, Josef Bacik wrote:
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1
> > /dev/mapper/dshelf1a
> > Couldn't read chunk tree
> > WTF???
> > ERROR: open ctree failed
> 
> That's new, the chunk tree wasn't failing before right?  Anyway I
> pushed a change, it should work now, thanks,

works better now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
FS_INFO IS 0x55d38ce312a0
parent transid verify failed on 13577821667328 wanted 1602089 found
1602242
parent transid verify failed on 13577821667328 wanted 1602089 found
1602242
parent transid verify failed on 13577821667328 wanted 1602089 found
1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found
1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found
1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found
1602242
Ignoring transid failure
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x55d38ce312a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
Ignoring transid failure
Ignoring transid failure
(...)
many similar lines, now it's running for a while, I'll update when it's
done.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 18:36                                                                                                               ` Josef Bacik
@ 2022-04-05 19:51                                                                                                                 ` Marc MERLIN
  2022-04-05 19:56                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05 19:51 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 02:36:29PM -0400, Josef Bacik wrote:
> > Couldn't read chunk tree
> > WTF???
> > ERROR: open ctree failed
> 
> That's new, the chunk tree wasn't failing before right?  Anyway I
> pushed a change, it should work now, thanks,

It failed for one commands we did before:
 ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a
btrfs-progs v5.16.2
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Couldn't read chunk tree

trfs-progs v5.16.2
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
Couldn't find the last root for 4
Couldn't setup device tree
ERROR: unable to open /dev/mapper/dshelf1a

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 19:51                                                                                                                 ` Marc MERLIN
@ 2022-04-05 19:56                                                                                                                   ` Josef Bacik
  2022-04-05 19:59                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05 19:56 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 5, 2022 at 3:51 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Apr 05, 2022 at 02:36:29PM -0400, Josef Bacik wrote:
> > > Couldn't read chunk tree
> > > WTF???
> > > ERROR: open ctree failed
> >
> > That's new, the chunk tree wasn't failing before right?  Anyway I
> > pushed a change, it should work now, thanks,
>
> It failed for one commands we did before:
>  ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a
> btrfs-progs v5.16.2
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> Couldn't read chunk tree
>

Ok I think this is from you redirecting into your device, can you do

btrfs inspect-internal dump-super -s 0
btrfs inspect-internal dump-super -s 1

and see if they're different?  We may have to put your old super back.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 19:56                                                                                                                   ` Josef Bacik
@ 2022-04-05 19:59                                                                                                                     ` Marc MERLIN
  2022-04-05 20:05                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05 19:59 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 03:56:31PM -0400, Josef Bacik wrote:
> On Tue, Apr 5, 2022 at 3:51 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Tue, Apr 05, 2022 at 02:36:29PM -0400, Josef Bacik wrote:
> > > > Couldn't read chunk tree
> > > > WTF???
> > > > ERROR: open ctree failed
> > >
> > > That's new, the chunk tree wasn't failing before right?  Anyway I
> > > pushed a change, it should work now, thanks,
> >
> > It failed for one commands we did before:
> >  ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a
> > btrfs-progs v5.16.2
> > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > Couldn't read chunk tree
> >
> 
> Ok I think this is from you redirecting into your device, can you do
 
Sorry for being an idiot.

> btrfs inspect-internal dump-super -s 0
> btrfs inspect-internal dump-super -s 1
> 
> and see if they're different?  We may have to put your old super back.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-super -s 0 /dev/mapper/dshelf1a > /tmp/0
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-super -s 1 /dev/mapper/dshelf1a > /tmp/1
gargamel:/var/local/src/btrfs-progs-josefbacik# diff -u /tmp/[01]
--- /tmp/0	2022-04-05 12:58:11.757530345 -0700
+++ /tmp/1	2022-04-05 12:58:14.769452304 -0700
@@ -1,27 +1,27 @@
-superblock: bytenr=65536, device=/dev/mapper/dshelf1a
+superblock: bytenr=67108864, device=/dev/mapper/dshelf1a
 ---------------------------------------------------------
 csum_type		0 (crc32c)
 csum_size		4
-csum			0xd3d00183 [match]
-bytenr			65536
+csum			0x70d015f7 [match]
+bytenr			67108864
 flags			0x1
 			( WRITTEN )
 magic			_BHRfS_M [match]
 fsid			96539b8c-ccc9-47bf-9e6c-29305890941e
 metadata_uuid		96539b8c-ccc9-47bf-9e6c-29305890941e
 label			dshelf1
-generation		1602089
-root			13577814573056
+generation		1602298
+root			15645253091328
 sys_array_size		129
-chunk_root_generation	1600938
+chunk_root_generation	1602203
 root_level		1
-chunk_root		21069824
+chunk_root		21479424
 chunk_root_level	1
 log_root		0
 log_root_transid	0
 log_root_level		0
 total_bytes		24004156973056
-bytes_used		15113376952320
+bytes_used		15144801386496
 sectorsize		4096
 nodesize		16384
 leafsize (deprecated)	16384
@@ -36,13 +36,13 @@
 			  BIG_METADATA |
 			  EXTENDED_IREF |
 			  SKINNY_METADATA )
-cache_generation	1602089
-uuid_tree_generation	1602089
+cache_generation	1602298
+uuid_tree_generation	1602298
 dev_item.uuid		8d4b0f25-0de9-47a6-a993-bdd301287f30
 dev_item.fsid		96539b8c-ccc9-47bf-9e6c-29305890941e [match]
 dev_item.type		0
 dev_item.total_bytes	24004156973056
-dev_item.bytes_used	15178439589888
+dev_item.bytes_used	15184882040832
 dev_item.io_align	4096
 dev_item.io_width	4096
 dev_item.sector_size	4096

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 19:59                                                                                                                     ` Marc MERLIN
@ 2022-04-05 20:05                                                                                                                       ` Josef Bacik
  2022-04-05 20:08                                                                                                                         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05 20:05 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 5, 2022 at 3:59 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Apr 05, 2022 at 03:56:31PM -0400, Josef Bacik wrote:
> > On Tue, Apr 5, 2022 at 3:51 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Tue, Apr 05, 2022 at 02:36:29PM -0400, Josef Bacik wrote:
> > > > > Couldn't read chunk tree
> > > > > WTF???
> > > > > ERROR: open ctree failed
> > > >
> > > > That's new, the chunk tree wasn't failing before right?  Anyway I
> > > > pushed a change, it should work now, thanks,
> > >
> > > It failed for one commands we did before:
> > >  ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a
> > > btrfs-progs v5.16.2
> > > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > > Couldn't read chunk tree
> > >
> >
> > Ok I think this is from you redirecting into your device, can you do
>
> Sorry for being an idiot.
>
> > btrfs inspect-internal dump-super -s 0
> > btrfs inspect-internal dump-super -s 1
> >
> > and see if they're different?  We may have to put your old super back.  Thanks,
>

Well it's still the same, and this thing is 20mib into your fs so IDK
how it would be screwing up now.  Can you do

./btrfs inspect-internal dump-tree -b 21069824

and see what that spits out?  IDK why it would suddenly start
complaining about your chunk root.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 20:05                                                                                                                       ` Josef Bacik
@ 2022-04-05 20:08                                                                                                                         ` Marc MERLIN
  2022-04-05 20:24                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05 20:08 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 04:05:05PM -0400, Josef Bacik wrote:
> Well it's still the same, and this thing is 20mib into your fs so IDK
> how it would be screwing up now.  Can you do
> 
> ./btrfs inspect-internal dump-tree -b 21069824
> 
> and see what that spits out?  IDK why it would suddenly start
> complaining about your chunk root.  Thanks,

Thanks for your patience and sticking with me
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 21069824 /dev/mapper/dshelf1a >/dev/null
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
FS_INFO IS 0x557c43fde470
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Ignoring transid failure
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x557c43fde470
node 21069824 level 1 items 95 free space 398 generation 1600938 owner CHUNK_TREE
node 21069824 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
        key (DEV_ITEMS DEV_ITEM 1) block 21135360 gen 1600938
        key (FIRST_CHUNK_TREE CHUNK_ITEM 159480020992) block 21086208 gen 278
        key (FIRST_CHUNK_TREE CHUNK_ITEM 325910003712) block 21102592 gen 357
        key (FIRST_CHUNK_TREE CHUNK_ITEM 490729373696) block 21118976 gen 450
        key (FIRST_CHUNK_TREE CHUNK_ITEM 657159356416) block 21184512 gen 593
        key (FIRST_CHUNK_TREE CHUNK_ITEM 821978726400) block 21938176 gen 1453956
        key (FIRST_CHUNK_TREE CHUNK_ITEM 986798096384) block 21151744 gen 696
        key (FIRST_CHUNK_TREE CHUNK_ITEM 1153228079104) block 21020672 gen 773
        key (FIRST_CHUNK_TREE CHUNK_ITEM 1319658061824) block 21250048 gen 846
        key (FIRST_CHUNK_TREE CHUNK_ITEM 1484477431808) block 21266432 gen 921
        key (FIRST_CHUNK_TREE CHUNK_ITEM 1650907414528) block 21282816 gen 1027
        key (FIRST_CHUNK_TREE CHUNK_ITEM 1817337397248) block 21168128 gen 1106
        key (FIRST_CHUNK_TREE CHUNK_ITEM 1982156767232) block 21037056 gen 1170
        key (FIRST_CHUNK_TREE CHUNK_ITEM 2148586749952) block 21233664 gen 1231
        key (FIRST_CHUNK_TREE CHUNK_ITEM 2315016732672) block 20971520 gen 1293
        key (FIRST_CHUNK_TREE CHUNK_ITEM 2479836102656) block 21315584 gen 1346
        key (FIRST_CHUNK_TREE CHUNK_ITEM 2646266085376) block 21381120 gen 1394
        key (FIRST_CHUNK_TREE CHUNK_ITEM 2812696068096) block 20987904 gen 1443
        key (FIRST_CHUNK_TREE CHUNK_ITEM 2977515438080) block 21348352 gen 1502
        key (FIRST_CHUNK_TREE CHUNK_ITEM 3143945420800) block 21364736 gen 1557
        key (FIRST_CHUNK_TREE CHUNK_ITEM 3310375403520) block 21397504 gen 1608
        key (FIRST_CHUNK_TREE CHUNK_ITEM 3475194773504) block 21463040 gen 1649
        key (FIRST_CHUNK_TREE CHUNK_ITEM 3641624756224) block 21004288 gen 1692
        key (FIRST_CHUNK_TREE CHUNK_ITEM 3808054738944) block 21200896 gen 1732
        key (FIRST_CHUNK_TREE CHUNK_ITEM 3972874108928) block 21446656 gen 1774
        key (FIRST_CHUNK_TREE CHUNK_ITEM 4139304091648) block 21053440 gen 1822
        key (FIRST_CHUNK_TREE CHUNK_ITEM 4305734074368) block 21544960 gen 1868
        key (FIRST_CHUNK_TREE CHUNK_ITEM 4470553444352) block 21299200 gen 1597466
        key (FIRST_CHUNK_TREE CHUNK_ITEM 4636983427072) block 21577728 gen 1965
        key (FIRST_CHUNK_TREE CHUNK_ITEM 4803413409792) block 21413888 gen 2014
        key (FIRST_CHUNK_TREE CHUNK_ITEM 4968232779776) block 21528576 gen 2065
        key (FIRST_CHUNK_TREE CHUNK_ITEM 5134662762496) block 21626880 gen 2118
        key (FIRST_CHUNK_TREE CHUNK_ITEM 5301092745216) block 21561344 gen 2167
        key (FIRST_CHUNK_TREE CHUNK_ITEM 5465912115200) block 21594112 gen 2218
        key (FIRST_CHUNK_TREE CHUNK_ITEM 5632342097920) block 21430272 gen 2270
        key (FIRST_CHUNK_TREE CHUNK_ITEM 5798772080640) block 21692416 gen 2315
        key (FIRST_CHUNK_TREE CHUNK_ITEM 5963591450624) block 21643264 gen 2358
        key (FIRST_CHUNK_TREE CHUNK_ITEM 6130021433344) block 21659648 gen 2411
        key (FIRST_CHUNK_TREE CHUNK_ITEM 6294840803328) block 21676032 gen 2465
        key (FIRST_CHUNK_TREE CHUNK_ITEM 6461270786048) block 21757952 gen 2512
        key (FIRST_CHUNK_TREE CHUNK_ITEM 6627700768768) block 21495808 gen 2558
        key (FIRST_CHUNK_TREE CHUNK_ITEM 6792520138752) block 21708800 gen 2607
        key (FIRST_CHUNK_TREE CHUNK_ITEM 6958950121472) block 21807104 gen 2655
        key (FIRST_CHUNK_TREE CHUNK_ITEM 7125380104192) block 21725184 gen 2711
        key (FIRST_CHUNK_TREE CHUNK_ITEM 7290199474176) block 21512192 gen 2762
        key (FIRST_CHUNK_TREE CHUNK_ITEM 7456629456896) block 21774336 gen 2808
        key (FIRST_CHUNK_TREE CHUNK_ITEM 7623059439616) block 21872640 gen 1463345
        key (FIRST_CHUNK_TREE CHUNK_ITEM 7787878809600) block 21889024 gen 2907
        key (FIRST_CHUNK_TREE CHUNK_ITEM 7954308792320) block 21905408 gen 2956
        key (FIRST_CHUNK_TREE CHUNK_ITEM 8120738775040) block 21790720 gen 3004
        key (FIRST_CHUNK_TREE CHUNK_ITEM 8285558145024) block 21839872 gen 3057
        key (FIRST_CHUNK_TREE CHUNK_ITEM 8451988127744) block 21856256 gen 3107
        key (FIRST_CHUNK_TREE CHUNK_ITEM 8618418110464) block 21921792 gen 3156
        key (FIRST_CHUNK_TREE CHUNK_ITEM 8783237480448) block 21217280 gen 3205
        key (FIRST_CHUNK_TREE CHUNK_ITEM 8949667463168) block 22003712 gen 3253
        key (FIRST_CHUNK_TREE CHUNK_ITEM 9116097445888) block 21954560 gen 3304
        key (FIRST_CHUNK_TREE CHUNK_ITEM 9280916815872) block 22036480 gen 3357
        key (FIRST_CHUNK_TREE CHUNK_ITEM 9447346798592) block 22020096 gen 1463291
        key (FIRST_CHUNK_TREE CHUNK_ITEM 9613776781312) block 21987328 gen 3462
        key (FIRST_CHUNK_TREE CHUNK_ITEM 9778596151296) block 22413312 gen 1463210
        key (FIRST_CHUNK_TREE CHUNK_ITEM 9945026134016) block 22052864 gen 3565
        key (FIRST_CHUNK_TREE CHUNK_ITEM 10111456116736) block 22085632 gen 1463201
        key (FIRST_CHUNK_TREE CHUNK_ITEM 10276275486720) block 21823488 gen 1463183
        key (FIRST_CHUNK_TREE CHUNK_ITEM 10442705469440) block 21331968 gen 1463138
        key (FIRST_CHUNK_TREE CHUNK_ITEM 10609135452160) block 22118400 gen 3761
        key (FIRST_CHUNK_TREE CHUNK_ITEM 10773954822144) block 22315008 gen 1463057
        key (FIRST_CHUNK_TREE CHUNK_ITEM 10940384804864) block 22151168 gen 1463003
        key (FIRST_CHUNK_TREE CHUNK_ITEM 11106814787584) block 22495232 gen 1572124
        key (FIRST_CHUNK_TREE CHUNK_ITEM 11271634157568) block 22200320 gen 1462940
        key (FIRST_CHUNK_TREE CHUNK_ITEM 11438064140288) block 22282240 gen 1462904
        key (FIRST_CHUNK_TREE CHUNK_ITEM 11607715348480) block 22249472 gen 1462571
        key (FIRST_CHUNK_TREE CHUNK_ITEM 11770387234816) block 22265856 gen 1462454
        key (FIRST_CHUNK_TREE CHUNK_ITEM 11948091506688) block 22167552 gen 1462274
        key (FIRST_CHUNK_TREE CHUNK_ITEM 12136600305664) block 22183936 gen 1462103
        key (FIRST_CHUNK_TREE CHUNK_ITEM 12282629193728) block 22134784 gen 1461923
        key (FIRST_CHUNK_TREE CHUNK_ITEM 12451240214528) block 22347776 gen 1357546
        key (FIRST_CHUNK_TREE CHUNK_ITEM 12619817680896) block 22069248 gen 504747
        key (FIRST_CHUNK_TREE CHUNK_ITEM 12794871152640) block 22364160 gen 1404175
        key (FIRST_CHUNK_TREE CHUNK_ITEM 12961301135360) block 22298624 gen 1423748
        key (FIRST_CHUNK_TREE CHUNK_ITEM 13176183717888) block 21741568 gen 1234369
        key (FIRST_CHUNK_TREE CHUNK_ITEM 13357713195008) block 22380544 gen 1332045
        key (FIRST_CHUNK_TREE CHUNK_ITEM 13560717508608) block 22396928 gen 1441271
        key (FIRST_CHUNK_TREE CHUNK_ITEM 13735838089216) block 22429696 gen 1423550
        key (FIRST_CHUNK_TREE CHUNK_ITEM 13904415555584) block 22102016 gen 1461914
        key (FIRST_CHUNK_TREE CHUNK_ITEM 14103091347456) block 21970944 gen 1461905
        key (FIRST_CHUNK_TREE CHUNK_ITEM 14271702368256) block 22462464 gen 1479229
        key (FIRST_CHUNK_TREE CHUNK_ITEM 14443501060096) block 22528000 gen 1572115
        key (FIRST_CHUNK_TREE CHUNK_ITEM 14648585748480) block 22233088 gen 1571944
        key (FIRST_CHUNK_TREE CHUNK_ITEM 14823605665792) block 22446080 gen 1571791
        key (FIRST_CHUNK_TREE CHUNK_ITEM 14990035648512) block 22544384 gen 1556078
        key (FIRST_CHUNK_TREE CHUNK_ITEM 15156465631232) block 22511616 gen 1555799
        key (FIRST_CHUNK_TREE CHUNK_ITEM 15322895613952) block 22577152 gen 1586277
        key (FIRST_CHUNK_TREE CHUNK_ITEM 15489325596672) block 22478848 gen 1561557
        key (FIRST_CHUNK_TREE CHUNK_ITEM 15664345513984) block 22593536 gen 1590219
        key (FIRST_CHUNK_TREE CHUNK_ITEM 15847955365888) block 22216704 gen 1600938
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 20:08                                                                                                                         ` Marc MERLIN
@ 2022-04-05 20:24                                                                                                                           ` Josef Bacik
  2022-04-05 20:37                                                                                                                             ` Marc MERLIN
  2022-04-05 23:51                                                                                                                             ` Zygo Blaxell
  0 siblings, 2 replies; 479+ messages in thread
From: Josef Bacik @ 2022-04-05 20:24 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 5, 2022 at 4:08 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Apr 05, 2022 at 04:05:05PM -0400, Josef Bacik wrote:
> > Well it's still the same, and this thing is 20mib into your fs so IDK
> > how it would be screwing up now.  Can you do
> >
> > ./btrfs inspect-internal dump-tree -b 21069824
> >
> > and see what that spits out?  IDK why it would suddenly start
> > complaining about your chunk root.  Thanks,
>
> Thanks for your patience and sticking with me
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 21069824 /dev/mapper/dshelf1a >/dev/null

Ok well that worked, which means it found the chunk tree fine, I'm
going to chalk this up to it just fucking with me and ignore it for
now.  I pushed some changes for the find root thing, can you re-run

./btrfs-find-root -o 1 /dev/whatever

it should be less noisy and spit out one line at the end, "Found tree
root at blah blah".  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 20:24                                                                                                                           ` Josef Bacik
@ 2022-04-05 20:37                                                                                                                             ` Marc MERLIN
  2022-04-05 21:07                                                                                                                               ` Josef Bacik
  2022-04-05 23:51                                                                                                                             ` Zygo Blaxell
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05 20:37 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 04:24:16PM -0400, Josef Bacik wrote:
> On Tue, Apr 5, 2022 at 4:08 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Tue, Apr 05, 2022 at 04:05:05PM -0400, Josef Bacik wrote:
> > > Well it's still the same, and this thing is 20mib into your fs so IDK
> > > how it would be screwing up now.  Can you do
> > >
> > > ./btrfs inspect-internal dump-tree -b 21069824
> > >
> > > and see what that spits out?  IDK why it would suddenly start
> > > complaining about your chunk root.  Thanks,
> >
> > Thanks for your patience and sticking with me
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 21069824 /dev/mapper/dshelf1a >/dev/null
> 
> Ok well that worked, which means it found the chunk tree fine, I'm
> going to chalk this up to it just fucking with me and ignore it for
> now.  I pushed some changes for the find root thing, can you re-run


Typo I fixed:
kernel-shared/disk-io.c: In function ‘verify_parent_transid’:
kernel-shared/disk-io.c:278:35: error: ‘struct btrfs_fs_info’ has no member named ‘suppress_check_block_errrs’; did you mean ‘suppress_check_block_errors’?
  278 |   if (eb->fs_info && eb->fs_info->suppress_check_block_errrs)
      |                                   ^~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                   suppress_check_block_errors

> ./btrfs-find-root -o 1 /dev/whatever
> 
> it should be less noisy and spit out one line at the end, "Found tree
> root at blah blah".  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# l btrfs-find-root
-rwxr-xr-x 1 root staff 3375608 Apr  5 13:29 btrfs-find-root*
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x56289ce172a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x56289ce172a0
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
(many deleted)
Ignoring transid failure
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
Found tree root at 13577814573056 gen 1602089 level 1

Good news :)
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 20:37                                                                                                                             ` Marc MERLIN
@ 2022-04-05 21:07                                                                                                                               ` Josef Bacik
  2022-04-05 21:14                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05 21:07 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 5, 2022 at 4:37 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Apr 05, 2022 at 04:24:16PM -0400, Josef Bacik wrote:
> > On Tue, Apr 5, 2022 at 4:08 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Tue, Apr 05, 2022 at 04:05:05PM -0400, Josef Bacik wrote:
> > > > Well it's still the same, and this thing is 20mib into your fs so IDK
> > > > how it would be screwing up now.  Can you do
> > > >
> > > > ./btrfs inspect-internal dump-tree -b 21069824
> > > >
> > > > and see what that spits out?  IDK why it would suddenly start
> > > > complaining about your chunk root.  Thanks,
> > >
> > > Thanks for your patience and sticking with me
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 21069824 /dev/mapper/dshelf1a >/dev/null
> >
> > Ok well that worked, which means it found the chunk tree fine, I'm
> > going to chalk this up to it just fucking with me and ignore it for
> > now.  I pushed some changes for the find root thing, can you re-run
>
>
> Typo I fixed:
> kernel-shared/disk-io.c: In function ‘verify_parent_transid’:
> kernel-shared/disk-io.c:278:35: error: ‘struct btrfs_fs_info’ has no member named ‘suppress_check_block_errrs’; did you mean ‘suppress_check_block_errors’?
>   278 |   if (eb->fs_info && eb->fs_info->suppress_check_block_errrs)
>       |                                   ^~~~~~~~~~~~~~~~~~~~~~~~~~
>       |                                   suppress_check_block_errors
>
> > ./btrfs-find-root -o 1 /dev/whatever
> >
> > it should be less noisy and spit out one line at the end, "Found tree
> > root at blah blah".  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# l btrfs-find-root
> -rwxr-xr-x 1 root staff 3375608 Apr  5 13:29 btrfs-find-root*
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> FS_INFO IS 0x56289ce172a0
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Couldn't find the last root for 4
> Couldn't setup device tree
> FS_INFO AFTER IS 0x56289ce172a0
> Ignoring transid failure
> Ignoring transid failure
> Ignoring transid failure
> Ignoring transid failure
> (many deleted)
> Ignoring transid failure
> Superblock thinks the generation is 1602089
> Superblock thinks the level is 1
> Found tree root at 13577814573056 gen 1602089 level 1
>
> Good news :)

Heh, except it's because I forgot to tell it to treat transid
mismatch's as a real problem.  Added some more code, lets give that a
whirl, thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 21:07                                                                                                                               ` Josef Bacik
@ 2022-04-05 21:14                                                                                                                                 ` Marc MERLIN
  2022-04-05 21:19                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05 21:14 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 05:07:27PM -0400, Josef Bacik wrote:
> Heh, except it's because I forgot to tell it to treat transid
> mismatch's as a real problem.  Added some more code, lets give that a
> whirl, thanks,

Looks better

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x559090e392a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x559090e392a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
Found tree root at 13577814573056 gen 1602089 level 1


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 21:14                                                                                                                                 ` Marc MERLIN
@ 2022-04-05 21:19                                                                                                                                   ` Josef Bacik
  2022-04-05 21:25                                                                                                                                     ` Marc MERLIN
  2022-04-05 21:26                                                                                                                                     ` Marc MERLIN
  0 siblings, 2 replies; 479+ messages in thread
From: Josef Bacik @ 2022-04-05 21:19 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 5, 2022 at 5:14 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Apr 05, 2022 at 05:07:27PM -0400, Josef Bacik wrote:
> > Heh, except it's because I forgot to tell it to treat transid
> > mismatch's as a real problem.  Added some more code, lets give that a
> > whirl, thanks,
>
> Looks better
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> FS_INFO IS 0x559090e392a0
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Couldn't find the last root for 4
> Couldn't setup device tree
> FS_INFO AFTER IS 0x559090e392a0
> Superblock thinks the generation is 1602089
> Superblock thinks the level is 1
> Found tree root at 13577814573056 gen 1602089 level 1

Otra vez por favor,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 21:19                                                                                                                                   ` Josef Bacik
@ 2022-04-05 21:25                                                                                                                                     ` Marc MERLIN
  2022-04-05 21:26                                                                                                                                     ` Marc MERLIN
  1 sibling, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05 21:25 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 05:19:57PM -0400, Josef Bacik wrote:
> Otra vez por favor,

sauron:/var/local/src/btrfs-progs-josefbacik# make
    [CC]     btrfs-find-root.o
btrfs-find-root.c:381:13: warning: ‘print_find_root_result’ defined but not used [-Wunused-function]
  381 | static void print_find_root_result(struct cache_tree *result,


I commented out a couple of unused functions, built and

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x556c1c6bb2a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x556c1c6bb2a0
gargamel:/var/local/src/btrfs-progs-josefbacik#
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 21:19                                                                                                                                   ` Josef Bacik
  2022-04-05 21:25                                                                                                                                     ` Marc MERLIN
@ 2022-04-05 21:26                                                                                                                                     ` Marc MERLIN
  2022-04-05 21:35                                                                                                                                       ` Josef Bacik
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05 21:26 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 05:19:57PM -0400, Josef Bacik wrote:
> Otra vez por favor,

(gdb) run -o 1 /dev/mapper/dshelf1a
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs-find-root -o 1 /dev/mapper/dshelf1a
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x5555555cf2a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x5555555cf2a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1

Program received signal SIGSEGV, Segmentation fault.
lookup_cache_extent (tree=tree@entry=0x5555555cf308, start=start@entry=505282560, size=size@entry=16384) at common/extent-cache.c:142
142		range.start = start;
(gdb) bt
#0  lookup_cache_extent (tree=tree@entry=0x5555555cf308, start=start@entry=505282560, size=size@entry=16384)
    at common/extent-cache.c:142
#1  0x0000555555579d1e in alloc_extent_buffer (fs_info=0x5555555cf2a0, bytenr=505282560, blocksize=16384)
    at kernel-shared/extent_io.c:737
#2  0x000055555556e44c in btrfs_find_create_tree_block (fs_info=<optimized out>, bytenr=<optimized out>)
    at kernel-shared/disk-io.c:230
#3  0x0000555555570208 in read_tree_block (fs_info=0x5555555cf2a0, bytenr=<optimized out>, parent_transid=1601370)
    at kernel-shared/disk-io.c:374
#4  0x00005555555a06df in try_read_block (slot=0, eb=0x5555721ef1a0) at btrfs-find-root.c:109
#5  count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:162
#6  0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164
#7  0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164
#8  0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164
#9  0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164
#10 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164
#11 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 21:26                                                                                                                                     ` Marc MERLIN
@ 2022-04-05 21:35                                                                                                                                       ` Josef Bacik
  2022-04-05 21:43                                                                                                                                         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05 21:35 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 5, 2022 at 5:26 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Apr 05, 2022 at 05:19:57PM -0400, Josef Bacik wrote:
> > Otra vez por favor,
>
> (gdb) run -o 1 /dev/mapper/dshelf1a
> Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs-find-root -o 1 /dev/mapper/dshelf1a
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> FS_INFO IS 0x5555555cf2a0
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Couldn't find the last root for 4
> Couldn't setup device tree
> FS_INFO AFTER IS 0x5555555cf2a0
> Superblock thinks the generation is 1602089
> Superblock thinks the level is 1
>
> Program received signal SIGSEGV, Segmentation fault.

Ugh sorry, try again.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 21:35                                                                                                                                       ` Josef Bacik
@ 2022-04-05 21:43                                                                                                                                         ` Marc MERLIN
  2022-04-05 22:41                                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05 21:43 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 05:35:02PM -0400, Josef Bacik wrote:
> On Tue, Apr 5, 2022 at 5:26 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Tue, Apr 05, 2022 at 05:19:57PM -0400, Josef Bacik wrote:
> > > Otra vez por favor,
> >
> > (gdb) run -o 1 /dev/mapper/dshelf1a
> > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs-find-root -o 1 /dev/mapper/dshelf1a
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > FS_INFO IS 0x5555555cf2a0
> > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > Couldn't find the last root for 4
> > Couldn't setup device tree
> > FS_INFO AFTER IS 0x5555555cf2a0
> > Superblock thinks the generation is 1602089
> > Superblock thinks the level is 1
> >
> > Program received signal SIGSEGV, Segmentation fault.
> 
> Ugh sorry, try again.  Thanks,


Thanks. Note that you still have 2 functions to comment out
From https://github.com/josefbacik/btrfs-progs
   ce32ea15..914c9847  for-marc   -> origin/for-marc
Updating ce32ea15..914c9847
Fast-forward
 btrfs-find-root.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
sauron:/var/local/src/btrfs-progs-josefbacik# make
    [CC]     btrfs-find-root.o
btrfs-find-root.c:383:13: warning: ‘print_find_root_result’ defined but not used [-Wunused-function]
  383 | static void print_find_root_result(struct cache_tree *result,


New version after I fixed the build warnings that were errors for me
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x564897ec32a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x564897ec32a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
Found tree root at 13577814573056 gen 1602089 level 1
gargamel:/var/local/src/btrfs-progs-josefbacik# 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 16:22                                                                                                 ` Roman Mamedov
@ 2022-04-05 22:06                                                                                                   ` Marc MERLIN
  0 siblings, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05 22:06 UTC (permalink / raw)
  To: Roman Mamedov; +Cc: Josef Bacik, linux-btrfs

On Tue, Apr 05, 2022 at 09:22:30PM +0500, Roman Mamedov wrote:
> On Mon, 4 Apr 2022 18:42:59 -0700
> Marc MERLIN <marc@merlins.org> wrote:
> 
> > And shit, I got distracted and sent the text output to
> > /dev/mapper/dshelf1a, so I clobbered about 30K of the device. 
> > I'm assuming there was probably something there?
> > 
> > Script started on Mon 04 Apr 2022 18:36:51 PDT
> > 1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m ./btrfs-inspect-internal dump-tree -b 13577821667328 &>/dev/mapper/dshelf1a
> > btrfs inspect-internal dump-tree: not enough arguments: 0 but at least 1 expected
> 
> It doesn't appear like it would write 30K to the device, it failed right away
> since you didn't *specify* the device; the above output was to STDERR, and
> there would be no writes to STDOUT (i.e. the device). Did you check the
> content of /dev/mapper/dshelf1a (with less -f, for example)?

Yeah, good point.
I checked it and it only has one line clobbered:
btrfs inspect-internal dump-tree: not enough arguments: 0 but at least 1 expected

then it's filled with 0's.

So I lost only the first sector, and it's not been used. All good. 
(that happened in another command as like you said the command I pasted
couldn't have done this)

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 21:43                                                                                                                                         ` Marc MERLIN
@ 2022-04-05 22:41                                                                                                                                           ` Josef Bacik
  2022-04-05 22:58                                                                                                                                             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-05 22:41 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 5, 2022 at 5:43 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Apr 05, 2022 at 05:35:02PM -0400, Josef Bacik wrote:
> > On Tue, Apr 5, 2022 at 5:26 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Tue, Apr 05, 2022 at 05:19:57PM -0400, Josef Bacik wrote:
> > > > Otra vez por favor,
> > >
> > > (gdb) run -o 1 /dev/mapper/dshelf1a
> > > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs-find-root -o 1 /dev/mapper/dshelf1a
> > > [Thread debugging using libthread_db enabled]
> > > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > > parent transid verify failed on 22216704 wanted 1600938 found 1602177
> > > FS_INFO IS 0x5555555cf2a0
> > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> > > Couldn't find the last root for 4
> > > Couldn't setup device tree
> > > FS_INFO AFTER IS 0x5555555cf2a0
> > > Superblock thinks the generation is 1602089
> > > Superblock thinks the level is 1
> > >
> > > Program received signal SIGSEGV, Segmentation fault.
> >
> > Ugh sorry, try again.  Thanks,
>
>
> Thanks. Note that you still have 2 functions to comment out
> From https://github.com/josefbacik/btrfs-progs
>    ce32ea15..914c9847  for-marc   -> origin/for-marc
> Updating ce32ea15..914c9847
> Fast-forward
>  btrfs-find-root.c | 8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)
> sauron:/var/local/src/btrfs-progs-josefbacik# make
>     [CC]     btrfs-find-root.o
> btrfs-find-root.c:383:13: warning: ‘print_find_root_result’ defined but not used [-Wunused-function]
>   383 | static void print_find_root_result(struct cache_tree *result,
>
>
> New version after I fixed the build warnings that were errors for me
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> FS_INFO IS 0x564897ec32a0
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
> Couldn't find the last root for 4
> Couldn't setup device tree
> FS_INFO AFTER IS 0x564897ec32a0
> Superblock thinks the generation is 1602089
> Superblock thinks the level is 1
> Found tree root at 13577814573056 gen 1602089 level 1
> gargamel:/var/local/src/btrfs-progs-josefbacik#
>

I'm wandering down this rabbit hole because if I'm able to read the
blocks fine, wtf is the device root not being found.  I've pushed more
printfs, lets see what that says.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 22:41                                                                                                                                           ` Josef Bacik
@ 2022-04-05 22:58                                                                                                                                             ` Marc MERLIN
  2022-04-06  0:23                                                                                                                                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-05 22:58 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 1872 bytes --]

On Tue, Apr 05, 2022 at 06:41:02PM -0400, Josef Bacik wrote:
> I'm wandering down this rabbit hole because if I'm able to read the
> blocks fine, wtf is the device root not being found.  I've pushed more
> printfs, lets see what that says.  Thanks,
 
Sure thing. Output is kind of big, summary here, full output bzipped and
attached
But isn't the tree found at the end?
Found tree root at 13577814573056 gen 1602089 level 1

output:
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x55be0733a2a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x55be0733a2a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
checking root item 11221
bad root
checking root item 11222
checking root item 11223
bad root
checking root item 11224
checking root item 11322
checking root item 11322
bad root
checking root item 152935
(...)
checking block 13577814573056
bad block
checking block 13577814573056
bad block
checking block 13577814573056
bad block
checking block 13577814573056
checking block 13577814573056
bad block
Found tree root at 13577814573056 gen 1602089 level 1


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

[-- Attachment #2: out.bz2 --]
[-- Type: application/octet-stream, Size: 16464 bytes --]

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 20:24                                                                                                                           ` Josef Bacik
  2022-04-05 20:37                                                                                                                             ` Marc MERLIN
@ 2022-04-05 23:51                                                                                                                             ` Zygo Blaxell
  2022-04-06  0:08                                                                                                                               ` Marc MERLIN
  1 sibling, 1 reply; 479+ messages in thread
From: Zygo Blaxell @ 2022-04-05 23:51 UTC (permalink / raw)
  To: Josef Bacik; +Cc: Marc MERLIN, linux-btrfs

On Tue, Apr 05, 2022 at 04:24:16PM -0400, Josef Bacik wrote:
> On Tue, Apr 5, 2022 at 4:08 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Tue, Apr 05, 2022 at 04:05:05PM -0400, Josef Bacik wrote:
> > > Well it's still the same, and this thing is 20mib into your fs so IDK
> > > how it would be screwing up now.  Can you do
> > >
> > > ./btrfs inspect-internal dump-tree -b 21069824
> > >
> > > and see what that spits out?  IDK why it would suddenly start
> > > complaining about your chunk root.  Thanks,
> >
> > Thanks for your patience and sticking with me
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 21069824 /dev/mapper/dshelf1a >/dev/null
> 
> Ok well that worked, which means it found the chunk tree fine, I'm

There's two copies of the chunk tree in dup metadata.  Maybe one of them
is bad?

It seems surprising to me if that's the case.  I didn't think the
userspace read code did anything like the current->pid % 2 dance,
and even if it did, I'd expect it to show up before now.

Other possibilites include irreproducible reads coming from the bcache
layer if that's still active (e.g. if it was on mdadm raid1, and the
raid1 mirrors were out of sync and didn't know it).  Hopefully not,
because it would make this...challenging.

> going to chalk this up to it just fucking with me and ignore it for
> now.  I pushed some changes for the find root thing, can you re-run
> 
> ./btrfs-find-root -o 1 /dev/whatever
> 
> it should be less noisy and spit out one line at the end, "Found tree
> root at blah blah".  Thanks,
> 
> Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 23:51                                                                                                                             ` Zygo Blaxell
@ 2022-04-06  0:08                                                                                                                               ` Marc MERLIN
  2022-04-06  1:40                                                                                                                                 ` Zygo Blaxell
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-06  0:08 UTC (permalink / raw)
  To: Zygo Blaxell; +Cc: Josef Bacik, linux-btrfs

On Tue, Apr 05, 2022 at 07:51:29PM -0400, Zygo Blaxell wrote:
> Other possibilites include irreproducible reads coming from the bcache
> layer if that's still active (e.g. if it was on mdadm raid1, and the

bcache is still active, correct.
mdadm5 array is running with 4 drives out of 5.

I'm not sure if I full understand the tree root issue we've been having.

We know from the dump that we have these backups
backup_tree_root:       13577814573056  gen: 1602089    level: 1
backup_tree_root:       13577775284224  gen: 1602086    level: 1
backup_tree_root:       13577821667328  gen: 1602087    level: 1
backup_tree_root:       13577779511296  gen: 1602088    level: 1

but they didn't seem to work, or are we sure that none of them don't?

Then we did a custom scan, but it found one of the 4 backup_tree_root we
already had, so that doesn't help, correct?
Found tree root at 13577814573056 gen 1602089 level 1

Was the expectation that there are more backup_tree_root hiding
somewhere and we've been trying to find them?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-05 22:58                                                                                                                                             ` Marc MERLIN
@ 2022-04-06  0:23                                                                                                                                               ` Josef Bacik
  2022-04-06  0:30                                                                                                                                                 ` Marc MERLIN
  2022-04-06  0:35                                                                                                                                                 ` Marc MERLIN
  0 siblings, 2 replies; 479+ messages in thread
From: Josef Bacik @ 2022-04-06  0:23 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 5, 2022 at 6:58 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Apr 05, 2022 at 06:41:02PM -0400, Josef Bacik wrote:
> > I'm wandering down this rabbit hole because if I'm able to read the
> > blocks fine, wtf is the device root not being found.  I've pushed more
> > printfs, lets see what that says.  Thanks,
>
> Sure thing. Output is kind of big, summary here, full output bzipped and
> attached
> But isn't the tree found at the end?
> Found tree root at 13577814573056 gen 1602089 level 1
>

Alright it's time for the crazy.  Go ahead and pull and start running.
This is going to take a while to run, we're basically going to walk
and check all the node pointers in the tree root, if it doesn't look
right we're going to search the metadata for the best copy to use, and
then update the block to point at the new block.  It has to do the
full search every time, because we don't have time for me to properly
implement a cache, so don't be worried if it takes a while.

It may print out stuff, if it looks like it's looping stop it and let
me know, but I don't think I fucked it up.  You're going to see a lot
"fixed root <id>", "fixed slot <whatever>", if you see it repeating
the same slot or root then we know we have a problem.

Once it's done can you do the print-tree ROOT thing again so I can see
if it did the right thing.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06  0:23                                                                                                                                               ` Josef Bacik
@ 2022-04-06  0:30                                                                                                                                                 ` Marc MERLIN
  2022-04-06  0:35                                                                                                                                                 ` Marc MERLIN
  1 sibling, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-04-06  0:30 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 08:23:42PM -0400, Josef Bacik wrote:
> On Tue, Apr 5, 2022 at 6:58 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Tue, Apr 05, 2022 at 06:41:02PM -0400, Josef Bacik wrote:
> > > I'm wandering down this rabbit hole because if I'm able to read the
> > > blocks fine, wtf is the device root not being found.  I've pushed more
> > > printfs, lets see what that says.  Thanks,
> >
> > Sure thing. Output is kind of big, summary here, full output bzipped and
> > attached
> > But isn't the tree found at the end?
> > Found tree root at 13577814573056 gen 1602089 level 1
> >
> 
> Alright it's time for the crazy.  Go ahead and pull and start running.
> This is going to take a while to run, we're basically going to walk
> and check all the node pointers in the tree root, if it doesn't look
> right we're going to search the metadata for the best copy to use, and
> then update the block to point at the new block.  It has to do the
> full search every time, because we don't have time for me to properly
> implement a cache, so don't be worried if it takes a while.

Cool, thanks again for all your help, I hope it helps improving the tools
and recovery for all

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outn
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x562612a192a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x562612a192a0

I'll let it run and report back.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06  0:23                                                                                                                                               ` Josef Bacik
  2022-04-06  0:30                                                                                                                                                 ` Marc MERLIN
@ 2022-04-06  0:35                                                                                                                                                 ` Marc MERLIN
  2022-04-06  0:39                                                                                                                                                   ` Josef Bacik
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-06  0:35 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 08:23:42PM -0400, Josef Bacik wrote:
> Alright it's time for the crazy.  Go ahead and pull and start running.
> This is going to take a while to run, we're basically going to walk
> and check all the node pointers in the tree root, if it doesn't look
> right we're going to search the metadata for the best copy to use, and
> then update the block to point at the new block.  It has to do the
> full search every time, because we don't have time for me to properly
> implement a cache, so don't be worried if it takes a while.
> 
> It may print out stuff, if it looks like it's looping stop it and let
> me know, but I don't think I fucked it up.  You're going to see a lot
> "fixed root <id>", "fixed slot <whatever>", if you see it repeating
> the same slot or root then we know we have a problem.
 
I assume this is not good?
ERROR: failed to write bytenr 13577814573056 length 16384 devid 1 dev_bytenr 13439981355008: Operation not permitted

I am running as root of course.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06  0:35                                                                                                                                                 ` Marc MERLIN
@ 2022-04-06  0:39                                                                                                                                                   ` Josef Bacik
  2022-04-06  1:08                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-06  0:39 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 5, 2022 at 8:35 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Apr 05, 2022 at 08:23:42PM -0400, Josef Bacik wrote:
> > Alright it's time for the crazy.  Go ahead and pull and start running.
> > This is going to take a while to run, we're basically going to walk
> > and check all the node pointers in the tree root, if it doesn't look
> > right we're going to search the metadata for the best copy to use, and
> > then update the block to point at the new block.  It has to do the
> > full search every time, because we don't have time for me to properly
> > implement a cache, so don't be worried if it takes a while.
> >
> > It may print out stuff, if it looks like it's looping stop it and let
> > me know, but I don't think I fucked it up.  You're going to see a lot
> > "fixed root <id>", "fixed slot <whatever>", if you see it repeating
> > the same slot or root then we know we have a problem.
>
> I assume this is not good?
> ERROR: failed to write bytenr 13577814573056 length 16384 devid 1 dev_bytenr 13439981355008: Operation not permitted
>
> I am running as root of course.
>

Yeah I'm just a dumbass, try again please. Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06  0:39                                                                                                                                                   ` Josef Bacik
@ 2022-04-06  1:08                                                                                                                                                     ` Josef Bacik
  2022-04-06  1:14                                                                                                                                                       ` Marc MERLIN
  2022-04-06  3:12                                                                                                                                                       ` Marc MERLIN
  0 siblings, 2 replies; 479+ messages in thread
From: Josef Bacik @ 2022-04-06  1:08 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 5, 2022 at 8:39 PM Josef Bacik <josef@toxicpanda.com> wrote:
>
> On Tue, Apr 5, 2022 at 8:35 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Tue, Apr 05, 2022 at 08:23:42PM -0400, Josef Bacik wrote:
> > > Alright it's time for the crazy.  Go ahead and pull and start running.
> > > This is going to take a while to run, we're basically going to walk
> > > and check all the node pointers in the tree root, if it doesn't look
> > > right we're going to search the metadata for the best copy to use, and
> > > then update the block to point at the new block.  It has to do the
> > > full search every time, because we don't have time for me to properly
> > > implement a cache, so don't be worried if it takes a while.
> > >
> > > It may print out stuff, if it looks like it's looping stop it and let
> > > me know, but I don't think I fucked it up.  You're going to see a lot
> > > "fixed root <id>", "fixed slot <whatever>", if you see it repeating
> > > the same slot or root then we know we have a problem.
> >
> > I assume this is not good?
> > ERROR: failed to write bytenr 13577814573056 length 16384 devid 1 dev_bytenr 13439981355008: Operation not permitted
> >
> > I am running as root of course.
> >
>
> Yeah I'm just a dumbass, try again please. Thanks,
>

Also keep in mind this isn't the final fix, this is the pre-repair so
hopefully fsck can put the rest of it back together.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06  1:08                                                                                                                                                     ` Josef Bacik
@ 2022-04-06  1:14                                                                                                                                                       ` Marc MERLIN
  2022-04-06  3:12                                                                                                                                                       ` Marc MERLIN
  1 sibling, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-04-06  1:14 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 09:08:32PM -0400, Josef Bacik wrote:
> > Yeah I'm just a dumbass, try again please. Thanks,

You've kindly given me around 20 fixes/patches in barely a day on a
filesystem that is not local to you :)

> Also keep in mind this isn't the final fix, this is the pre-repair so
> hopefully fsck can put the rest of it back together.  Thanks,

Ack.
It seems to work better or at least I got no output as of right now with
more runtime than last time. It'll likely run overnight and I'll report
tomorrow, thanks.

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outn
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x557e9cc375f0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x557e9cc375f0

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06  0:08                                                                                                                               ` Marc MERLIN
@ 2022-04-06  1:40                                                                                                                                 ` Zygo Blaxell
  2022-04-06  4:09                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Zygo Blaxell @ 2022-04-06  1:40 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Josef Bacik, linux-btrfs

On Tue, Apr 05, 2022 at 05:08:44PM -0700, Marc MERLIN wrote:
> On Tue, Apr 05, 2022 at 07:51:29PM -0400, Zygo Blaxell wrote:
> > Other possibilites include irreproducible reads coming from the bcache
> > layer if that's still active (e.g. if it was on mdadm raid1, and the
> 
> bcache is still active, correct.
> mdadm5 array is running with 4 drives out of 5.
> 
> I'm not sure if I full understand the tree root issue we've been having.

Based on the history, I'd expect the filesystem is missing some number
of tree nodes, from a few dozen to thousands, depending on how many
writes were dropped after the 2nd drive failure before it was detected.
Since the array was also degraded at that time, with 4 drives in raid5,
there's 3 data drives, and if one of them was offline then we'd have a 2/3
success rate reading metadata blocks and 1/3 garbage.  That's definitely
in the "we need to write new software to recover from this" territory.

Normally, I'd expect that once we dig through a few layers of simple
dropped write blocks, we'll start hitting metadata pages with bad csums
and trashed contents, since the parity blocks will be garbage in the raid5
stripes where the writes were lost.  One important data point against
this theory is that we have not seen a csum failure yet, so maybe this
is a different (possibly better) scenario.  Possibly some of the lost
writes on the raid5 are still stored in the bcache, so there's few or no
garbage blocks (though reading the array through the cache might evict
the last copy of usable data and make some damage permanent--you might
want to make a backup copy of the cache device).

Backup roots only work if writes are dropped only in the most recent
transaction, maybe two, because only these trees are guaranteed to be
intact on disk.  After that, previously occupied pages are fair game
for new write allocations, and old metadata will be lost.  Unlike other
filesystems, btrfs never writes metadata in the same place twice, so when
a write is dropped, there isn't an old copy of the data still available at
the location of the dropped write--that location contains some completely
unrelated piece of the metadata tree whose current version now lives
at some other location.  Later tree updates will overwrite old copies
of the updated page, destroying the data in the affected page forever.
Essentially there will be a set of metadata pages where you have two
versions of different ages, and another set of metadata pages where you
have zero versions, and (hopefully) most of the other pages are intact.

> We know from the dump that we have these backups
> backup_tree_root:       13577814573056  gen: 1602089    level: 1
> backup_tree_root:       13577775284224  gen: 1602086    level: 1
> backup_tree_root:       13577821667328  gen: 1602087    level: 1
> backup_tree_root:       13577779511296  gen: 1602088    level: 1
> 
> but they didn't seem to work, or are we sure that none of them don't?

The root pointers are just pointers--having a pointer to a tree root isn't
going to help if the trees were never written at the referenced location.
Recursively, if you have a tree node but you don't have the next-level
nodes the previous-level node points to, then the root isn't useful
because large parts of its tree are unreachable.  If you're missing a
leaf node then whatever data was in the leaf is gone.  Once you've found
a root node, you can try walking over the entire tree to see if it's all
there (e.g. with btrfs ins dump-tree -t N), but if anything's missing
in any part of the tree, then that tree root can't be used as-is.

> Then we did a custom scan, but it found one of the 4 backup_tree_root we
> already had, so that doesn't help, correct?
> Found tree root at 13577814573056 gen 1602089 level 1
> 
> Was the expectation that there are more backup_tree_root hiding
> somewhere and we've been trying to find them?

If we have a superblock, the chunk tree, and a subvol tree, we can
drop all the other trees and rebuild them (bonus points if the csum
tree survived, then we can verify all the data was recovered correctly;
otherwise, we can read all the files and make a new csum tree but it won't
detect any data corruption that might have happened in degraded mode).
This is roughly what 'btrfs check --init-extent-tree' does (though due
to implementation details it has a few extra dependencies that might
get in the way) and you can find subvol roots with btrfs-find-root.

If we don't have any intact subvol trees (or the subvol trees we really
want aren't intact), then we can't recover this way.  Instead we'd have
to scrape the disk looking for metadata leaf nodes, and try to reinsert
those into a new tree structure.  The trick here is that we'll have the
duplicated and inconsitent nodes and we won't have some nodes at all,
and we'll have to make sense of those (or pass it to the existing btrfs
check and hope it can cope with them).  I'm guessing that a simplified
version of this is what Josef is building at this point, or will be
building soon if we aren't extremely lucky and find an intact subvol tree.
After building an intact subvol tree (even with a few garbage items in it
as long as check can handle them) we can go back to the --init-extent-tree
step and rebuild the rest of the filesystem.

> Thanks,
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>  
> Home page: http://marc.merlins.org/  
> 

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06  1:08                                                                                                                                                     ` Josef Bacik
  2022-04-06  1:14                                                                                                                                                       ` Marc MERLIN
@ 2022-04-06  3:12                                                                                                                                                       ` Marc MERLIN
  2022-04-06  3:34                                                                                                                                                         ` Marc MERLIN
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-06  3:12 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 09:08:32PM -0400, Josef Bacik wrote:
> Also keep in mind this isn't the final fix, this is the pre-repair so
> hopefully fsck can put the rest of it back together.  Thanks,

Done, so now I can run 
btrfs check --repair /dev/mapper/dshelf1a ?

with or without mode=lowmem?

Do I need to reset the ctree too, or repair will do it?
[642321.755553] BTRFS warning (device dm-17): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
[642321.755566] BTRFS info (device dm-17): trying to use backup root at mount time
[642321.755568] BTRFS info (device dm-17): disk space caching is enabled
[642321.755569] BTRFS info (device dm-17): has skinny extents
[642322.242161] BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
[642322.282393] BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
[642322.313951] BTRFS error (device dm-17): failed to read chunk tree: -5
[642322.339490] BTRFS error (device dm-17): open_ctree failed


gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outn
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x557e9cc375f0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x557e9cc375f0
fixed slot 0
fixed slot 2
fixed slot 5
fixed slot 7
(...)
Couldn't find a replacement block for slot 261
fixed slot 263
fixed slot 264
fixed slot 265
fixed slot 267
fixed slot 269
fixed slot 271
fixed slot 273
fixed slot 274
fixed slot 276
fixed slot 277
fixed slot 278
fixed slot 279
fixed slot 282
fixed slot 283
fixed slot 285
fixed slot 286
fixed slot 287
fixed slot 288
fixed slot 289
fixed slot 291
fixed slot 293
fixed slot 294
fixed slot 296
fixed slot 297
fixed slot 298
fixed slot 300
fixed slot 301
fixed slot 302
fixed slot 303
fixed slot 304
fixed slot 305
fixed slot 306
fixed slot 307
fixed slot 308
fixed slot 309
fixed slot 311
fixed slot 312
fixed slot 313
fixed slot 314
fixed slot 315
fixed slot 316
fixed slot 317
fixed slot 318
fixed slot 319
fixed slot 320
fixed slot 321


Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06  3:12                                                                                                                                                       ` Marc MERLIN
@ 2022-04-06  3:34                                                                                                                                                         ` Marc MERLIN
  2022-04-06 15:20                                                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-06  3:34 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 05, 2022 at 08:12:55PM -0700, Marc MERLIN wrote:
> On Tue, Apr 05, 2022 at 09:08:32PM -0400, Josef Bacik wrote:
> > Also keep in mind this isn't the final fix, this is the pre-repair so
> > hopefully fsck can put the rest of it back together.  Thanks,
> 
> Done, so now I can run 
> btrfs check --repair /dev/mapper/dshelf1a ?
 
Here is regular check:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x564d665f1e50
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
parent transid verify failed on 15645261971456 wanted 1602297 found 1600989
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to read block groups: Input/output error
FS_INFO AFTER IS 0x564d665f1e50
Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
parent transid verify failed on 15645261971456 wanted 1602297 found 1600989
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to repair root items: Input/output error
[2/7] checking extents
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 13577702703104 wanted 1602089 found 1602241
parent transid verify failed on 13577702703104 wanted 1602089 found 1602241
parent transid verify failed on 13577702703104 wanted 1602089 found 1602241
parent transid verify failed on 12511683117056 wanted 1602089 found 1602202
parent transid verify failed on 12511683117056 wanted 1602089 found 1602202
parent transid verify failed on 12511683117056 wanted 1602089 found 1602202
parent transid verify failed on 15645121789952 wanted 1602089 found 1602135
parent transid verify failed on 15645121789952 wanted 1602089 found 1600922
parent transid verify failed on 15645121789952 wanted 1602089 found 1602135
parent transid verify failed on 15645274947584 wanted 1602089 found 1602298
parent transid verify failed on 15645274947584 wanted 1602089 found 1602298
parent transid verify failed on 15645274947584 wanted 1602089 found 1602298
checksum verify failed on 13577870934016 wanted 0xfffb4d18 found 0xb0be1c94
parent transid verify failed on 11160884019200 wanted 1602089 found 1602180
parent transid verify failed on 11160884019200 wanted 1602089 found 1602180
parent transid verify failed on 11160884019200 wanted 1602089 found 1602180
parent transid verify failed on 15645280632832 wanted 1602089 found 1602298
parent transid verify failed on 15645280632832 wanted 1602089 found 1601000
parent transid verify failed on 15645280632832 wanted 1602089 found 1602298
parent transid verify failed on 15645397942272 wanted 1602089 found 1602141
parent transid verify failed on 15645397942272 wanted 1602089 found 1602141
parent transid verify failed on 15645397942272 wanted 1602089 found 1602141
parent transid verify failed on 15645399678976 wanted 1602089 found 1602243
checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418
checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418
bad tree block 15645399678976, bad level, 236 > 8
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
ERROR: invalid generation for root 5, have 1602297 expect (0, 1602090]
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6
parent transid verify failed on 15645248897024 wanted 1602297 found 1600937
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6
bad tree block 15645248897024, bad level, 66 > 8
The following tree block(s) is corrupted in tree 5:
	tree block bytenr: 15645248880640, level: 2, node key: (256, 1, 0)
root 5 root dir 256 not found
parent transid verify failed on 13577702703104 wanted 1602089 found 1602241
parent transid verify failed on 12511683117056 wanted 1602089 found 1602202
parent transid verify failed on 15645121789952 wanted 1602089 found 1602135
parent transid verify failed on 15645274947584 wanted 1602089 found 1602298
parent transid verify failed on 11160884019200 wanted 1602089 found 1602180
parent transid verify failed on 15645280632832 wanted 1602089 found 1602298
parent transid verify failed on 15645397942272 wanted 1602089 found 1602141
parent transid verify failed on 15645399678976 wanted 1602089 found 1602243
checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418
checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418
bad tree block 15645399678976, bad level, 236 > 8
ERROR: errors found in fs roots
found 0 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 0
total fs tree bytes: 0
total extent tree bytes: 0
btree space waste bytes: 0
file data blocks allocated: 0
 referenced 0
gargamel:/var/local/src/btrfs-progs-josefbacik#


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06  1:40                                                                                                                                 ` Zygo Blaxell
@ 2022-04-06  4:09                                                                                                                                   ` Marc MERLIN
  2022-04-06 18:07                                                                                                                                     ` Zygo Blaxell
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-06  4:09 UTC (permalink / raw)
  To: Zygo Blaxell; +Cc: Josef Bacik, linux-btrfs

On Tue, Apr 05, 2022 at 09:40:51PM -0400, Zygo Blaxell wrote:
> Based on the history, I'd expect the filesystem is missing some number
> of tree nodes, from a few dozen to thousands, depending on how many
> writes were dropped after the 2nd drive failure before it was detected.
> Since the array was also degraded at that time, with 4 drives in raid5,
> there's 3 data drives, and if one of them was offline then we'd have a 2/3
> success rate reading metadata blocks and 1/3 garbage.  That's definitely
> in the "we need to write new software to recover from this" territory.
 
I think your conclusion is correct. I'm very dismayed that the
filesystem didn't go read only right away.
Hell, the mdadm block device should have done read only as soon as it
lost more than one drive.
Why were any writes allowed once more than one drive was missing?

Let's look at this for a second:
Mar 28 02:28:11 gargamel kernel: [1512988.446844] sd 6:1:8:0: Device offlined - not ready after error recovery
Mar 28 02:28:11 gargamel kernel: [1512988.475270] sd 6:1:8:0: rejecting I/O to offline device
Mar 28 02:28:11 gargamel kernel: [1512988.491531] blk_update_request: I/O error, dev sdi, sector 261928312 op 0x0:(READ) flags 0x84700
 phys_seg 42 prio class 0
Mar 28 02:28:11 gargamel kernel: [1512988.525073] blk_update_request: I/O error, dev sdi, sector 261928824 op 0x0:(READ) flags 0x80700
 phys_seg 5 prio class 0
Mar 28 02:28:12 gargamel kernel: [1512988.579667] blk_update_request: I/O error, dev sdi, sector 261927936 op 0x0:(READ) flags 0x80700
 phys_seg 47 prio class 0
(..)
Mar 28 02:28:12 gargamel kernel: [1512988.615910] md: super_written gets error=10
Mar 28 02:28:12 gargamel kernel: [1512988.619241] md/raid:md7: Disk failure on sdi1, disabling device.
Mar 28 02:28:12 gargamel kernel: [1512988.619241] md/raid:md7: Operation continuing on 4 devices.
Mar 28 02:28:21 gargamel kernel: [1512998.170192] usb 2-1.6-port1: disabled by hub (EMI?), re-enabling...
Mar 28 02:28:21 gargamel kernel: [1512998.240404] print_req_error: 134 callbacks suppressed
Mar 28 02:28:21 gargamel kernel: [1512998.240406] blk_update_request: I/O error, dev sdi, sector 11721044992 op 0x0:(READ) flags 0x807
00 phys_seg 1 prio class 0
Mar 28 02:28:21 gargamel kernel: [1512998.243415] ftdi_sio 2-1.6.1:1.0: device disconnected
Mar 28 02:28:21 gargamel kernel: [1512998.341221] blk_update_request: I/O error, dev sdi, sector 11721044992 op 0x0:(READ) flags 0x0 p
(...)
Mar 28 02:28:22 gargamel kernel: [1512998.716351] blk_update_request: I/O error, dev sdi, sector 2058 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Mar 28 02:28:22 gargamel kernel: [1512998.716362] blk_update_request: I/O error, dev sdi, sector 2059 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0

Ok, one drive died, but raid5 continues degraded.
md7 : active raid5 sdi1[5](F) sdo1[7] sdg1[6] sdj1[3] sdh1[1]
      23441561600 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UUUU_]
      bitmap: 0/44 pages [0KB], 65536KB chunk

not sure what these were:
Mar 29 00:00:08 gargamel kernel: [1590505.415665] bcache: bch_count_backing_io_errors() md7: Read-ahead I/O failed on backing device, ignore
Mar 29 00:00:09 gargamel kernel: [1590505.866094] bcache: bch_count_backing_io_errors() md7: Read-ahead I/O failed on backing device, ignore

9H later a 2nd drive dies just when I'm replacing the failed one:
Mar 29 09:30:12 gargamel kernel: [1624709.301830] sd 6:1:5:0: [sdh] tag#523 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s
Mar 29 09:30:12 gargamel kernel: [1624709.331812] sd 6:1:5:0: [sdh] tag#523 CDB: Read(16) 88 00 00 00 00 00 00 26 3f d0 00 00 00 18 00 00
Mar 29 09:30:12 gargamel kernel: [1624709.359459] blk_update_request: I/O error, dev sdh, sector 2506704 op 0x0:(READ) flags 0x0 phys_seg 3 prio class 0
Mar 29 09:30:12 gargamel kernel: [1624709.359465] md/raid:md7: read error not correctable (sector 2504656 on sdh1).
Mar 29 09:30:12 gargamel kernel: [1624709.359471] md/raid:md7: read error not correctable (sector 2504664 on sdh1).
Mar 29 09:30:12 gargamel kernel: [1624709.359472] md/raid:md7: read error not correctable (sector 2504672 on sdh1).
Mar 29 09:30:12 gargamel kernel: [1624709.359486] md/raid:md7: read error not correctable (sector 2504656 on sdh1).
Mar 29 09:30:12 gargamel kernel: [1624709.455886] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.681637] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.695785] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.717624] md/raid:md7: read error not correctable (sector 2504664 on sdh1).
Mar 29 09:30:13 gargamel kernel: [1624709.739746] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.757206] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.770348] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.790298] md/raid:md7: read error not correctable (sector 2504672 on sdh1).
Mar 29 09:30:13 gargamel kernel: [1624709.812546] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.825856] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.839006] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.866656] md/raid:md7: read error not correctable (sector 24815552 on sdh1).
Mar 29 09:30:13 gargamel kernel: [1624709.898796] md/raid:md7: read error not correctable (sector 24815552 on sdh1).
Mar 29 09:30:13 gargamel kernel: [1624709.921135] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.934315] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.947500] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.988589] md/raid:md7: read error not correctable (sector 1763985936 on sdh1).
Mar 29 09:30:13 gargamel kernel: [1624710.036204] md/raid:md7: read error not correctable (sector 1763985936 on sdh1).
Mar 29 09:30:13 gargamel kernel: [1624710.059121] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624710.088858] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624710.102026] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624710.158830] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.096055] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:36:37 gargamel kernel: [1625094.122910] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
Mar 29 09:36:37 gargamel kernel: [1625094.153249] md/raid:md7: read error not correctable (sector 6562801616 on sdh1).
Mar 29 09:36:37 gargamel kernel: [1625094.176011] md/raid:md7: read error not correctable (sector 6562801624 on sdh1).
Mar 29 09:36:37 gargamel kernel: [1625094.223351] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.250628] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.263726] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.276989] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.290121] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.303267] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.325084] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.342083] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.355206] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.368394] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.383304] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.396423] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.409498] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:36:37 gargamel kernel: [1625094.436355] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
Mar 29 09:36:37 gargamel kernel: [1625094.466729] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:36:37 gargamel kernel: [1625094.493600] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 3, flush 0, corrupt 0, gen 0
Mar 29 09:36:37 gargamel kernel: [1625094.523998] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:36:38 gargamel kernel: [1625094.550938] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 4, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.066422] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.093309] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 5, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.124768] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.151651] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 6, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.182803] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.209677] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 7, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.239972] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.266862] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 8, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.297234] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.324094] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 9, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.354422] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.381286] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 10, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.411926] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.438770] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 11, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.469361] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.496269] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 12, flush 0, corrupt 0, gen 0
Mar 29 09:37:35 gargamel kernel: [1625151.527455] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:35 gargamel kernel: [1625151.554360] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 13, flush 0, corrupt 0, gen 0
Mar 29 09:37:35 gargamel kernel: [1625151.584963] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:35 gargamel kernel: [1625151.611842] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 14, flush 0, corrupt 0, gen 0

24 seconds are lost before btrfs notices anything, and then it seems to
continue without going read only for another full minute before things
stopped.

Why would the raid5 not go read only immediately after 2 drives are lost?

> Normally, I'd expect that once we dig through a few layers of simple
> dropped write blocks, we'll start hitting metadata pages with bad csums
> and trashed contents, since the parity blocks will be garbage in the raid5
> stripes where the writes were lost.  One important data point against
> this theory is that we have not seen a csum failure yet, so maybe this
> is a different (possibly better) scenario.  Possibly some of the lost
> writes on the raid5 are still stored in the bcache, so there's few or no
> garbage blocks (though reading the array through the cache might evict
> the last copy of usable data and make some damage permanent--you might
> want to make a backup copy of the cache device).
 
Interesting, thanks.

> Backup roots only work if writes are dropped only in the most recent
> transaction, maybe two, because only these trees are guaranteed to be
> intact on disk.  After that, previously occupied pages are fair game
> for new write allocations, and old metadata will be lost.  Unlike other
> filesystems, btrfs never writes metadata in the same place twice, so when
> a write is dropped, there isn't an old copy of the data still available at
> the location of the dropped write--that location contains some completely
> unrelated piece of the metadata tree whose current version now lives
> at some other location.  Later tree updates will overwrite old copies
> of the updated page, destroying the data in the affected page forever.
> Essentially there will be a set of metadata pages where you have two
> versions of different ages, and another set of metadata pages where you
> have zero versions, and (hopefully) most of the other pages are intact.
 
I see. It's definitely a lot more complex and much more likely to break when some
amount of recent writes get lost/corrupted.

> If we have a superblock, the chunk tree, and a subvol tree, we can
> drop all the other trees and rebuild them (bonus points if the csum
> tree survived, then we can verify all the data was recovered correctly;
> otherwise, we can read all the files and make a new csum tree but it won't
> detect any data corruption that might have happened in degraded mode).
> This is roughly what 'btrfs check --init-extent-tree' does (though due
> to implementation details it has a few extra dependencies that might
> get in the way) and you can find subvol roots with btrfs-find-root.
 
Got it, thanks.

> If we don't have any intact subvol trees (or the subvol trees we really
> want aren't intact), then we can't recover this way.  Instead we'd have
> to scrape the disk looking for metadata leaf nodes, and try to reinsert
> those into a new tree structure.  The trick here is that we'll have the
> duplicated and inconsitent nodes and we won't have some nodes at all,
> and we'll have to make sense of those (or pass it to the existing btrfs
> check and hope it can cope with them).  I'm guessing that a simplified
> version of this is what Josef is building at this point, or will be
> building soon if we aren't extremely lucky and find an intact subvol tree.
> After building an intact subvol tree (even with a few garbage items in it
> as long as check can handle them) we can go back to the --init-extent-tree
> step and rebuild the rest of the filesystem.

I see. In that case, I'm still happy to help, to help improve the tools, but if
I'm looking at some amount of non trivial loss/corruption, at soe point
I'll go back to backups, since they'll be more intact than this now
damaged filesystem.

Thanks for the detailed answers.
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06  3:34                                                                                                                                                         ` Marc MERLIN
@ 2022-04-06 15:20                                                                                                                                                           ` Josef Bacik
  2022-04-06 18:54                                                                                                                                                             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-06 15:20 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 5, 2022 at 11:34 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Apr 05, 2022 at 08:12:55PM -0700, Marc MERLIN wrote:
> > On Tue, Apr 05, 2022 at 09:08:32PM -0400, Josef Bacik wrote:
> > > Also keep in mind this isn't the final fix, this is the pre-repair so
> > > hopefully fsck can put the rest of it back together.  Thanks,
> >
> > Done, so now I can run
> > btrfs check --repair /dev/mapper/dshelf1a ?
>
> Here is regular check:
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a

Woo ok we're almost there, that's great.  Can you re-run the
btrfs-find-root and see what it spits out this time?  It should run
faster now that it fixed everything, I'm wondering what it wasn't able
to fix and if I should run the same check on all the roots.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06  4:09                                                                                                                                   ` Marc MERLIN
@ 2022-04-06 18:07                                                                                                                                     ` Zygo Blaxell
  2022-04-06 19:13                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Zygo Blaxell @ 2022-04-06 18:07 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Josef Bacik, linux-btrfs

On Tue, Apr 05, 2022 at 09:09:13PM -0700, Marc MERLIN wrote:
> On Tue, Apr 05, 2022 at 09:40:51PM -0400, Zygo Blaxell wrote:
> > Based on the history, I'd expect the filesystem is missing some number
> > of tree nodes, from a few dozen to thousands, depending on how many
> > writes were dropped after the 2nd drive failure before it was detected.
> > Since the array was also degraded at that time, with 4 drives in raid5,
> > there's 3 data drives, and if one of them was offline then we'd have a 2/3
> > success rate reading metadata blocks and 1/3 garbage.  That's definitely
> > in the "we need to write new software to recover from this" territory.
>  
> I think your conclusion is correct. I'm very dismayed that the
> filesystem didn't go read only right away.
> Hell, the mdadm block device should have done read only as soon as it
> lost more than one drive.
> Why were any writes allowed once more than one drive was missing?

> Let's look at this for a second:
> Mar 28 02:28:11 gargamel kernel: [1512988.446844] sd 6:1:8:0: Device offlined - not ready after error recovery
> Mar 28 02:28:11 gargamel kernel: [1512988.475270] sd 6:1:8:0: rejecting I/O to offline device
> Mar 28 02:28:11 gargamel kernel: [1512988.491531] blk_update_request: I/O error, dev sdi, sector 261928312 op 0x0:(READ) flags 0x84700
>  phys_seg 42 prio class 0
> Mar 28 02:28:11 gargamel kernel: [1512988.525073] blk_update_request: I/O error, dev sdi, sector 261928824 op 0x0:(READ) flags 0x80700
>  phys_seg 5 prio class 0
> Mar 28 02:28:12 gargamel kernel: [1512988.579667] blk_update_request: I/O error, dev sdi, sector 261927936 op 0x0:(READ) flags 0x80700
>  phys_seg 47 prio class 0
> (..)
> Mar 28 02:28:12 gargamel kernel: [1512988.615910] md: super_written gets error=10
> Mar 28 02:28:12 gargamel kernel: [1512988.619241] md/raid:md7: Disk failure on sdi1, disabling device.
> Mar 28 02:28:12 gargamel kernel: [1512988.619241] md/raid:md7: Operation continuing on 4 devices.
> Mar 28 02:28:21 gargamel kernel: [1512998.170192] usb 2-1.6-port1: disabled by hub (EMI?), re-enabling...
> Mar 28 02:28:21 gargamel kernel: [1512998.240404] print_req_error: 134 callbacks suppressed
> Mar 28 02:28:21 gargamel kernel: [1512998.240406] blk_update_request: I/O error, dev sdi, sector 11721044992 op 0x0:(READ) flags 0x807
> 00 phys_seg 1 prio class 0
> Mar 28 02:28:21 gargamel kernel: [1512998.243415] ftdi_sio 2-1.6.1:1.0: device disconnected
> Mar 28 02:28:21 gargamel kernel: [1512998.341221] blk_update_request: I/O error, dev sdi, sector 11721044992 op 0x0:(READ) flags 0x0 p
> (...)
> Mar 28 02:28:22 gargamel kernel: [1512998.716351] blk_update_request: I/O error, dev sdi, sector 2058 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> Mar 28 02:28:22 gargamel kernel: [1512998.716362] blk_update_request: I/O error, dev sdi, sector 2059 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> 
> Ok, one drive died, but raid5 continues degraded.
> md7 : active raid5 sdi1[5](F) sdo1[7] sdg1[6] sdj1[3] sdh1[1]
>       23441561600 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UUUU_]
>       bitmap: 0/44 pages [0KB], 65536KB chunk
> 
> not sure what these were:
> Mar 29 00:00:08 gargamel kernel: [1590505.415665] bcache: bch_count_backing_io_errors() md7: Read-ahead I/O failed on backing device, ignore
> Mar 29 00:00:09 gargamel kernel: [1590505.866094] bcache: bch_count_backing_io_errors() md7: Read-ahead I/O failed on backing device, ignore

Readahead can fail for a number of reasons that aren't real problems,
e.g. if it's necessary to swap or evict pages to do the readahead,
the kernel doesn't bother, and fails the readahead operation instead.
Another common case is lower layers of block IO stack, where a readahead
request can't be satisfied because of some geometry constraint specific
to one device in a multi-device LV.  e.g.  a drive is no longer present
so it can't receive block IO requests.

Normally readahead errors are ignored.  If there's a real problem,
the application will issue a normal read later on and the error will be
reported then.  Or the readahead might have been entirely speculative,
but the app never tries to read the data at all.

> 9H later a 2nd drive dies just when I'm replacing the failed one:
> Mar 29 09:30:12 gargamel kernel: [1624709.301830] sd 6:1:5:0: [sdh] tag#523 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s
> Mar 29 09:30:12 gargamel kernel: [1624709.331812] sd 6:1:5:0: [sdh] tag#523 CDB: Read(16) 88 00 00 00 00 00 00 26 3f d0 00 00 00 18 00 00
> Mar 29 09:30:12 gargamel kernel: [1624709.359459] blk_update_request: I/O error, dev sdh, sector 2506704 op 0x0:(READ) flags 0x0 phys_seg 3 prio class 0
> Mar 29 09:30:12 gargamel kernel: [1624709.359465] md/raid:md7: read error not correctable (sector 2504656 on sdh1).
> Mar 29 09:30:12 gargamel kernel: [1624709.359471] md/raid:md7: read error not correctable (sector 2504664 on sdh1).
> Mar 29 09:30:12 gargamel kernel: [1624709.359472] md/raid:md7: read error not correctable (sector 2504672 on sdh1).
> Mar 29 09:30:12 gargamel kernel: [1624709.359486] md/raid:md7: read error not correctable (sector 2504656 on sdh1).
> Mar 29 09:30:12 gargamel kernel: [1624709.455886] md: super_written gets error=10
> Mar 29 09:30:13 gargamel kernel: [1624709.681637] md: super_written gets error=10
> Mar 29 09:30:13 gargamel kernel: [1624709.695785] md: super_written gets error=10
> Mar 29 09:30:13 gargamel kernel: [1624709.717624] md/raid:md7: read error not correctable (sector 2504664 on sdh1).
> Mar 29 09:30:13 gargamel kernel: [1624709.739746] md: super_written gets error=10
> Mar 29 09:30:13 gargamel kernel: [1624709.757206] md: super_written gets error=10
> Mar 29 09:30:13 gargamel kernel: [1624709.770348] md: super_written gets error=10
> Mar 29 09:30:13 gargamel kernel: [1624709.790298] md/raid:md7: read error not correctable (sector 2504672 on sdh1).
> Mar 29 09:30:13 gargamel kernel: [1624709.812546] md: super_written gets error=10
> Mar 29 09:30:13 gargamel kernel: [1624709.825856] md: super_written gets error=10
> Mar 29 09:30:13 gargamel kernel: [1624709.839006] md: super_written gets error=10
> Mar 29 09:30:13 gargamel kernel: [1624709.866656] md/raid:md7: read error not correctable (sector 24815552 on sdh1).
> Mar 29 09:30:13 gargamel kernel: [1624709.898796] md/raid:md7: read error not correctable (sector 24815552 on sdh1).
> Mar 29 09:30:13 gargamel kernel: [1624709.921135] md: super_written gets error=10
> Mar 29 09:30:13 gargamel kernel: [1624709.934315] md: super_written gets error=10
> Mar 29 09:30:13 gargamel kernel: [1624709.947500] md: super_written gets error=10
> Mar 29 09:30:13 gargamel kernel: [1624709.988589] md/raid:md7: read error not correctable (sector 1763985936 on sdh1).
> Mar 29 09:30:13 gargamel kernel: [1624710.036204] md/raid:md7: read error not correctable (sector 1763985936 on sdh1).
> Mar 29 09:30:13 gargamel kernel: [1624710.059121] md: super_written gets error=10
> Mar 29 09:30:13 gargamel kernel: [1624710.088858] md: super_written gets error=10
> Mar 29 09:30:13 gargamel kernel: [1624710.102026] md: super_written gets error=10
> Mar 29 09:30:13 gargamel kernel: [1624710.158830] md: super_written gets error=10
> Mar 29 09:36:37 gargamel kernel: [1625094.096055] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
> Mar 29 09:36:37 gargamel kernel: [1625094.122910] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
> Mar 29 09:36:37 gargamel kernel: [1625094.153249] md/raid:md7: read error not correctable (sector 6562801616 on sdh1).
> Mar 29 09:36:37 gargamel kernel: [1625094.176011] md/raid:md7: read error not correctable (sector 6562801624 on sdh1).
> Mar 29 09:36:37 gargamel kernel: [1625094.223351] md: super_written gets error=10
> Mar 29 09:36:37 gargamel kernel: [1625094.250628] md: super_written gets error=10
> Mar 29 09:36:37 gargamel kernel: [1625094.263726] md: super_written gets error=10
> Mar 29 09:36:37 gargamel kernel: [1625094.276989] md: super_written gets error=10
> Mar 29 09:36:37 gargamel kernel: [1625094.290121] md: super_written gets error=10
> Mar 29 09:36:37 gargamel kernel: [1625094.303267] md: super_written gets error=10
> Mar 29 09:36:37 gargamel kernel: [1625094.325084] md: super_written gets error=10
> Mar 29 09:36:37 gargamel kernel: [1625094.342083] md: super_written gets error=10
> Mar 29 09:36:37 gargamel kernel: [1625094.355206] md: super_written gets error=10
> Mar 29 09:36:37 gargamel kernel: [1625094.368394] md: super_written gets error=10
> Mar 29 09:36:37 gargamel kernel: [1625094.383304] md: super_written gets error=10
> Mar 29 09:36:37 gargamel kernel: [1625094.396423] md: super_written gets error=10
> Mar 29 09:36:37 gargamel kernel: [1625094.409498] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
> Mar 29 09:36:37 gargamel kernel: [1625094.436355] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
> Mar 29 09:36:37 gargamel kernel: [1625094.466729] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
> Mar 29 09:36:37 gargamel kernel: [1625094.493600] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 3, flush 0, corrupt 0, gen 0
> Mar 29 09:36:37 gargamel kernel: [1625094.523998] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
> Mar 29 09:36:38 gargamel kernel: [1625094.550938] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 4, flush 0, corrupt 0, gen 0
> Mar 29 09:37:34 gargamel kernel: [1625151.066422] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
> Mar 29 09:37:34 gargamel kernel: [1625151.093309] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 5, flush 0, corrupt 0, gen 0
> Mar 29 09:37:34 gargamel kernel: [1625151.124768] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
> Mar 29 09:37:34 gargamel kernel: [1625151.151651] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 6, flush 0, corrupt 0, gen 0
> Mar 29 09:37:34 gargamel kernel: [1625151.182803] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
> Mar 29 09:37:34 gargamel kernel: [1625151.209677] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 7, flush 0, corrupt 0, gen 0
> Mar 29 09:37:34 gargamel kernel: [1625151.239972] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
> Mar 29 09:37:34 gargamel kernel: [1625151.266862] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 8, flush 0, corrupt 0, gen 0
> Mar 29 09:37:34 gargamel kernel: [1625151.297234] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
> Mar 29 09:37:34 gargamel kernel: [1625151.324094] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 9, flush 0, corrupt 0, gen 0
> Mar 29 09:37:34 gargamel kernel: [1625151.354422] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
> Mar 29 09:37:34 gargamel kernel: [1625151.381286] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 10, flush 0, corrupt 0, gen 0
> Mar 29 09:37:34 gargamel kernel: [1625151.411926] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
> Mar 29 09:37:34 gargamel kernel: [1625151.438770] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 11, flush 0, corrupt 0, gen 0
> Mar 29 09:37:34 gargamel kernel: [1625151.469361] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
> Mar 29 09:37:34 gargamel kernel: [1625151.496269] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 12, flush 0, corrupt 0, gen 0
> Mar 29 09:37:35 gargamel kernel: [1625151.527455] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
> Mar 29 09:37:35 gargamel kernel: [1625151.554360] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 13, flush 0, corrupt 0, gen 0
> Mar 29 09:37:35 gargamel kernel: [1625151.584963] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
> Mar 29 09:37:35 gargamel kernel: [1625151.611842] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 14, flush 0, corrupt 0, gen 0
> 
> 24 seconds are lost before btrfs notices anything, and then it seems to
> continue without going read only for another full minute before things
> stopped.
> 
> Why would the raid5 not go read only immediately after 2 drives are lost?

Once a degraded array has an IO failure, mdadm assumes you're in
a data-scraping recovery mode and simply passes errors through.
bcache does something similar when the backing store fails.

btrfs is the agent that has to stop attempting writes, and make sure
any transaction in progress doesn't get committed.  ext4 has similar
responsibility and implements its own force-read-only feature.

I note that btrfs is reporting only read errors here.

There's a possibility that the drive dropped the writes during the
bus reset at the start of the second drive failure.  If write caching
was enabled in the drive, and the drive has a firmware bug that drops
the write cache contents (or it's just failing hardware, i.e. the CPU
running the drive firmware is getting killed by electrical problems on the
controller board, causing both the bus drop and the loss of write cache
contents), then writes in the drive's cache could be lost _after_ mdadm,
bcache and btrfs had been told by the drive that they were completed.

If the failing drive also reorders cached writes across flush
commands, then we go directly to parent transid verify failed in btrfs.
Parent transid verification is designed to detect this exact firmware
failure mode, and it usually works as intended.  It's a pretty direct
and reliable signal that write ordering is broken in a lower level of
the storage stack, and must be fixed or disabled before trying again.

Even if a single drive doesn't reorder writes, multiple drives in
raid effectively reorder writes between drives as each drive has its
own distinct write queue.  A dropped write that would be harmless in a
single-device filesystem could be harmful in a multi-device array as the
non-failing drives will have writes that occurred after the lost writes
on the failing drive.  Normally mdadm enforces flush ordering across all
component devices so that this isn't a problem, but if you have optimistic
firmware and a drive failure after the flush command returns success,
the inter-drive ordering enforcement fails and the result is the same
as if an individual drive had a write/flush reordering bug.

Firmware bugs are fairly rare across drive models, but failing drive
models are vastly overrepresented in the field.  I've tested hundreds
of firmware revisions and only about 4% fail with bad write reordering.
The few firmwares that do fail are found in popular low-end consumer
drives and cheap enterprise drives (they're the same drive underneath,
just a different label and price), so there are a lot of bad drives in
the field.  Once we identify a bad firmware revision, we avoid buying
any more of it, disable write caching in the firmware on the drives
we already have, and try to find a workload for the drive that is less
sensitive to failure (or performance--fsync latency goes up when the
drive can't use its cache for writes).

Some years ago we did a fleetwide audit of bad firmware drives and
found about a third of our drives were bad.  We disabled write cache on
all affected drives and prioritized replacement of the affected models.
This stopped the worst btrfs problems we had been having up to that point.
It also stopped a lot of strange one-off application data corruptions we
had been chasing for years before on ext4/mdadm arrays and in postgresql
databases.  We'd assumed those were some kind of application bug, but
we never found an app bug, and the problem disappeared once we turned
off the write cache.

> > If we don't have any intact subvol trees (or the subvol trees we really
> > want aren't intact), then we can't recover this way.  Instead we'd have
> > to scrape the disk looking for metadata leaf nodes, and try to reinsert
> > those into a new tree structure.  The trick here is that we'll have the
> > duplicated and inconsitent nodes and we won't have some nodes at all,
> > and we'll have to make sense of those (or pass it to the existing btrfs
> > check and hope it can cope with them).  I'm guessing that a simplified
> > version of this is what Josef is building at this point, or will be
> > building soon if we aren't extremely lucky and find an intact subvol tree.
> > After building an intact subvol tree (even with a few garbage items in it
> > as long as check can handle them) we can go back to the --init-extent-tree
> > step and rebuild the rest of the filesystem.
> 
> I see. In that case, I'm still happy to help, to help improve the tools, but if
> I'm looking at some amount of non trivial loss/corruption, at soe point
> I'll go back to backups, since they'll be more intact than this now
> damaged filesystem.

Yeah, experimental recovery code is fun, but robust backups and a working
disaster recovery plan is usually better.  Even if the filesystem is
up and running again, I'd want to compare all the files against backups
because I'd trust nothing on a filesystem after fsck touched it.

On the other hand, if you get lucky and the filesystem isn't too badly
damaged, then comparing the data with backups will be more convenient
than starting over with a new filesystem.

> Thanks for the detailed answers.
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>  
> Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06 15:20                                                                                                                                                           ` Josef Bacik
@ 2022-04-06 18:54                                                                                                                                                             ` Marc MERLIN
  2022-04-06 18:57                                                                                                                                                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-06 18:54 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Apr 06, 2022 at 11:20:13AM -0400, Josef Bacik wrote:
> On Tue, Apr 5, 2022 at 11:34 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Tue, Apr 05, 2022 at 08:12:55PM -0700, Marc MERLIN wrote:
> > > On Tue, Apr 05, 2022 at 09:08:32PM -0400, Josef Bacik wrote:
> > > > Also keep in mind this isn't the final fix, this is the pre-repair so
> > > > hopefully fsck can put the rest of it back together.  Thanks,
> > >
> > > Done, so now I can run
> > > btrfs check --repair /dev/mapper/dshelf1a ?
> >
> > Here is regular check:
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a
> 
> Woo ok we're almost there, that's great.  Can you re-run the
> btrfs-find-root and see what it spits out this time?  It should run
> faster now that it fixed everything, I'm wondering what it wasn't able
> to fix and if I should run the same check on all the roots.  Thanks,

took a few hours to run, but it's done now. Looks like it fixed things a 2nd time?

Should I run 
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a
now?

output:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outm
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x55cf0ce435f0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x55cf0ce435f0
fixed slot 0
fixed slot 2
fixed slot 5
fixed slot 7
fixed slot 12
fixed slot 27
fixed slot 28
fixed slot 32
fixed slot 34
fixed slot 35
fixed slot 36
fixed slot 45
fixed slot 48
fixed slot 49
fixed slot 50
fixed slot 52
fixed slot 53
fixed slot 54
fixed slot 55
fixed slot 56
fixed slot 60
fixed slot 61
fixed slot 63
fixed slot 76
fixed slot 77
fixed slot 85
fixed slot 86
fixed slot 89
fixed slot 93
fixed slot 98
fixed slot 99
fixed slot 100
fixed slot 103
fixed slot 104
fixed slot 106
fixed slot 107
fixed slot 108
fixed slot 109
fixed slot 110
fixed slot 111
fixed slot 112
fixed slot 114
fixed slot 120
fixed slot 135
fixed slot 140
fixed slot 141
fixed slot 142
fixed slot 146
fixed slot 155
fixed slot 156
fixed slot 158
fixed slot 159
fixed slot 160
fixed slot 161
fixed slot 165
fixed slot 166
fixed slot 171
fixed slot 173
fixed slot 174
fixed slot 180
fixed slot 181
fixed slot 182
fixed slot 183
fixed slot 184
fixed slot 185
fixed slot 186
fixed slot 212
fixed slot 213
fixed slot 221
fixed slot 222
fixed slot 224
fixed slot 226
fixed slot 227
fixed slot 228
fixed slot 229
fixed slot 231
fixed slot 232
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 244
fixed slot 245
fixed slot 248
fixed slot 252
fixed slot 259
fixed slot 260
Couldn't find a replacement block for slot 261
fixed slot 263
fixed slot 264
fixed slot 265
fixed slot 267
fixed slot 269
fixed slot 271
fixed slot 273
fixed slot 274
fixed slot 276
fixed slot 277
fixed slot 278
fixed slot 279
fixed slot 282
fixed slot 283
fixed slot 285
fixed slot 286
fixed slot 287
fixed slot 288
fixed slot 289
fixed slot 291
fixed slot 293
fixed slot 294
fixed slot 296
fixed slot 297
fixed slot 298
fixed slot 300
fixed slot 301
fixed slot 302
fixed slot 303
fixed slot 304
fixed slot 305
fixed slot 306
fixed slot 307
fixed slot 308
fixed slot 309
fixed slot 311
fixed slot 312
fixed slot 313
fixed slot 314
fixed slot 315
fixed slot 316
fixed slot 317
fixed slot 318
fixed slot 319
fixed slot 320
fixed slot 321
gargamel:/var/local/src/btrfs-progs-josefbacik#
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06 18:54                                                                                                                                                             ` Marc MERLIN
@ 2022-04-06 18:57                                                                                                                                                               ` Josef Bacik
  2022-04-06 19:16                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-06 18:57 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Apr 6, 2022 at 2:54 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Apr 06, 2022 at 11:20:13AM -0400, Josef Bacik wrote:
> > On Tue, Apr 5, 2022 at 11:34 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Tue, Apr 05, 2022 at 08:12:55PM -0700, Marc MERLIN wrote:
> > > > On Tue, Apr 05, 2022 at 09:08:32PM -0400, Josef Bacik wrote:
> > > > > Also keep in mind this isn't the final fix, this is the pre-repair so
> > > > > hopefully fsck can put the rest of it back together.  Thanks,
> > > >
> > > > Done, so now I can run
> > > > btrfs check --repair /dev/mapper/dshelf1a ?
> > >
> > > Here is regular check:
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a
> >
> > Woo ok we're almost there, that's great.  Can you re-run the
> > btrfs-find-root and see what it spits out this time?  It should run
> > faster now that it fixed everything, I'm wondering what it wasn't able
> > to fix and if I should run the same check on all the roots.  Thanks,
>
> took a few hours to run, but it's done now. Looks like it fixed things a 2nd time?
>
> Should I run
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a
> now?
>

Yeah lets go for that, I saw some errors on your fs tree's earlier, I
may need to adapt this to fix that tree, or if it's a snapshot we can
just delete it.  We can burn that bridge when we get to it, thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06 18:07                                                                                                                                     ` Zygo Blaxell
@ 2022-04-06 19:13                                                                                                                                       ` Marc MERLIN
  2022-04-06 19:45                                                                                                                                         ` Zygo Blaxell
                                                                                                                                                           ` (2 more replies)
  0 siblings, 3 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-04-06 19:13 UTC (permalink / raw)
  To: Zygo Blaxell; +Cc: Josef Bacik, linux-btrfs

On Wed, Apr 06, 2022 at 02:07:47PM -0400, Zygo Blaxell wrote:
> Readahead can fail for a number of reasons that aren't real problems,

thanks.

> Once a degraded array has an IO failure, mdadm assumes you're in
> a data-scraping recovery mode and simply passes errors through.
> bcache does something similar when the backing store fails.
 
shouldn't it go read only also?
I haven't found a setting to tell it to do that if it's not the default.

> btrfs is the agent that has to stop attempting writes, and make sure
> any transaction in progress doesn't get committed.  ext4 has similar
> responsibility and implements its own force-read-only feature.

Agreed, but I like the defense in multiple layers approach
mdadm knows that any data going to be written is going to be incomplete
due to the 2nd missing drive, and there are few to no scenarios where 
continuing to write is a good thing.

> I note that btrfs is reporting only read errors here.

Yeah, me too. That seems very bad and likely the cause or the corruption
I got.
A double raid failure (I've had them before) should be almost a non
event. The first write after that should fail and that should be the end
of it.
This underlying problem makes me very uncomfortable.
As much as I'm super thankful for Josef's efforts, and ultimately it
will make btrfs recovery better, which is a win for all, my situation
shouldn't have happened from what I can tell.

> There's a possibility that the drive dropped the writes during the
> bus reset at the start of the second drive failure.  If write caching
> was enabled in the drive, and the drive has a firmware bug that drops
> the write cache contents (or it's just failing hardware, i.e. the CPU
> running the drive firmware is getting killed by electrical problems on the
> controller board, causing both the bus drop and the loss of write cache
> contents), then writes in the drive's cache could be lost _after_ mdadm,
> bcache and btrfs had been told by the drive that they were completed.

That's true, but I've seen btfrs remount read only before, and it didn't
there. Shouldn't hard IO errors immediately cause btrfs to go read only?

Write caching is on all those drives though:
gargamel:/var/local/src/btrfs-progs-josefbacik# hdparm -v -W /dev/sdh
/dev/sdh:
 multcount     = 16 (on)
 readonly      =  0 (off)
 readahead     = 256 (on)
 geometry      = 729601/255/63, sectors = 11721045168, start = 0
 write-caching =  1 (on)

I haven't heard that these drives have broken caching, but maybe they do?
Device Model:     ST6000VN0041-2EL11C
Serial Number:    ZA18TVFZ
LU WWN Device Id: 5 000c50 0a4d9b49c
Firmware Version: SC61
User Capacity:    6,001,175,126,016 bytes [6.00 TB]

> If the failing drive also reorders cached writes across flush
> commands, then we go directly to parent transid verify failed in btrfs.
> Parent transid verification is designed to detect this exact firmware
> failure mode, and it usually works as intended.  It's a pretty direct
> and reliable signal that write ordering is broken in a lower level of
> the storage stack, and must be fixed or disabled before trying again.
 
Agreed, I do want to fix the underlying problem here.
What I can do is

1) disable write caching on the drives

2) disable bcache by removing the caching device

3) change the cache mode
gargamel:/sys/block/md7/bcache# cat cache_mode 
[writethrough] writeback writearound none
writethrough should be safe, but I could use writearound instead

4) if I end up wiping my device, I can just remove the bcache layer altogether.

> Even if a single drive doesn't reorder writes, multiple drives in
> raid effectively reorder writes between drives as each drive has its
> own distinct write queue.  A dropped write that would be harmless in a

very good point.

> single-device filesystem could be harmful in a multi-device array as the
> non-failing drives will have writes that occurred after the lost writes
> on the failing drive.  Normally mdadm enforces flush ordering across all
> component devices so that this isn't a problem, but if you have optimistic
> firmware and a drive failure after the flush command returns success,
> the inter-drive ordering enforcement fails and the result is the same
> as if an individual drive had a write/flush reordering bug.

That's all fair, but it feels like FS kept writing way longer than it
was supposed to and that is what worries me the most.

> Some years ago we did a fleetwide audit of bad firmware drives and
> found about a third of our drives were bad.  We disabled write cache on

Looks like I just want to disable write caching then. Correctness beats
speed for sure.

> Yeah, experimental recovery code is fun, but robust backups and a working
> disaster recovery plan is usually better.  Even if the filesystem is
> up and running again, I'd want to compare all the files against backups
> because I'd trust nothing on a filesystem after fsck touched it.
 
That is totally my plan. Given the output that I'm seeing, I'll definitely do
at least a diff of all files between the backup and the array being recovered.
I might even do an rsync, forcing md5 checksums, but if I do that, it
will take days and longer than restoring the backup.

> On the other hand, if you get lucky and the filesystem isn't too badly
> damaged, then comparing the data with backups will be more convenient
> than starting over with a new filesystem.

Agreed.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06 18:57                                                                                                                                                               ` Josef Bacik
@ 2022-04-06 19:16                                                                                                                                                                 ` Marc MERLIN
  2022-04-06 19:53                                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-06 19:16 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Apr 06, 2022 at 02:57:03PM -0400, Josef Bacik wrote:
> Yeah lets go for that, I saw some errors on your fs tree's earlier, I
> may need to adapt this to fix that tree, or if it's a snapshot we can
> just delete it.  We can burn that bridge when we get to it, thanks,

Sounds good. Is mode=lowmem a thing of the past by the way?
(I remember that the old regular repair would eat up my 32GB of RAM and
take the machine down)

The current one failed quickly though

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a 
enabling repair mode
WARNING:

	Do not use --repair unless you are advised to do so by a developer
	or an experienced user, and then only after having accepted that no
	fsck can successfully repair all types of filesystem corruption. Eg.
	some software or hardware bugs can fatally damage a volume.
	The operation will start in 10 seconds.
	Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x55e2e75defc0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
parent transid verify failed on 15645261971456 wanted 1602297 found 1600989
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to read block groups: Input/output error
FS_INFO AFTER IS 0x55e2e75defc0
Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
parent transid verify failed on 15645261971456 wanted 1602297 found 1600989
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to repair root items: Input/output error


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06 19:13                                                                                                                                       ` Marc MERLIN
@ 2022-04-06 19:45                                                                                                                                         ` Zygo Blaxell
  2022-04-06 20:38                                                                                                                                         ` figuring out why transient double raid failure caused a fair amount of btrfs corruption Marc MERLIN
  2022-04-22 18:48                                                                                                                                         ` Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) Marc MERLIN
  2 siblings, 0 replies; 479+ messages in thread
From: Zygo Blaxell @ 2022-04-06 19:45 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Josef Bacik, linux-btrfs

On Wed, Apr 06, 2022 at 12:13:17PM -0700, Marc MERLIN wrote:
> On Wed, Apr 06, 2022 at 02:07:47PM -0400, Zygo Blaxell wrote:
> > Readahead can fail for a number of reasons that aren't real problems,
> 
> thanks.
> 
> > Once a degraded array has an IO failure, mdadm assumes you're in
> > a data-scraping recovery mode and simply passes errors through.
> > bcache does something similar when the backing store fails.
>  
> shouldn't it go read only also?
> I haven't found a setting to tell it to do that if it's not the default.

bcache in writethrough mode could leave cached blocks dirty as long as
the SSD completes the write.  It should be reporting the write errors
back to upper layers.

Whether it actually does...I don't know, I haven't run that kind of
stress test on bcache.  I usually pair bcache SSD 1:1 with HDD, so if
either the SSD or HDD fails, I scrub or replace them both as a single
logical btrfs device, which makes this kind of question mostly irrelevant.
Unfortunately this approach doesn't work with raid5--the striping cuts IO
requests into tiny fragments, so the cache will try to cache everything.

> > btrfs is the agent that has to stop attempting writes, and make sure
> > any transaction in progress doesn't get committed.  ext4 has similar
> > responsibility and implements its own force-read-only feature.
> 
> Agreed, but I like the defense in multiple layers approach
> mdadm knows that any data going to be written is going to be incomplete
> due to the 2nd missing drive, and there are few to no scenarios where 
> continuing to write is a good thing.

True, there's no way the parity would make any sense with two drives
in a stripe missing during an update.  So maybe there weren't any
writes going through mdadm at that point, and everything failed at the
drive-offline event.  Or writes completed on the working drives before
the failing drive's failure was detected, messing up the data there.
A typical drive bus timeout is a long time, and a lot of raid stripes
could be modified on other drives while that happens.

> > There's a possibility that the drive dropped the writes during the
> > bus reset at the start of the second drive failure.  If write caching
> > was enabled in the drive, and the drive has a firmware bug that drops
> > the write cache contents (or it's just failing hardware, i.e. the CPU
> > running the drive firmware is getting killed by electrical problems on the
> > controller board, causing both the bus drop and the loss of write cache
> > contents), then writes in the drive's cache could be lost _after_ mdadm,
> > bcache and btrfs had been told by the drive that they were completed.
> 
> That's true, but I've seen btfrs remount read only before, and it didn't
> there. Shouldn't hard IO errors immediately cause btrfs to go read only?

No, only hard write IO errors on all metadata mirror drives, and cases
where btrfs needs to CoW a page or read free space tree, and can't find
an intact mirror.  Anything less is correctable (write failure on some
mirrors) or can be retried (any read failure) if the raid profile has
redundancy.

> Write caching is on all those drives though:
> gargamel:/var/local/src/btrfs-progs-josefbacik# hdparm -v -W /dev/sdh
> /dev/sdh:
>  multcount     = 16 (on)
>  readonly      =  0 (off)
>  readahead     = 256 (on)
>  geometry      = 729601/255/63, sectors = 11721045168, start = 0
>  write-caching =  1 (on)
> 
> I haven't heard that these drives have broken caching, but maybe they do?
> Device Model:     ST6000VN0041-2EL11C
> Serial Number:    ZA18TVFZ
> LU WWN Device Id: 5 000c50 0a4d9b49c
> Firmware Version: SC61
> User Capacity:    6,001,175,126,016 bytes [6.00 TB]

I've heard of problems with SC60, but IIRC SC61 was supposed to be the
fix for them.  I had some SC60 drives for a while without encountering
issues, but I didn't have any bus timeouts with them.  When they failed,
they had a burst of 10k+ UNC sectors and then stopped spinning up,
with no bus timeouts.

> > If the failing drive also reorders cached writes across flush
> > commands, then we go directly to parent transid verify failed in btrfs.
> > Parent transid verification is designed to detect this exact firmware
> > failure mode, and it usually works as intended.  It's a pretty direct
> > and reliable signal that write ordering is broken in a lower level of
> > the storage stack, and must be fixed or disabled before trying again.
>  
> Agreed, I do want to fix the underlying problem here.
> What I can do is
> 
> 1) disable write caching on the drives
> 
> 2) disable bcache by removing the caching device
> 
> 3) change the cache mode
> gargamel:/sys/block/md7/bcache# cat cache_mode 
> [writethrough] writeback writearound none
> writethrough should be safe, but I could use writearound instead
> 
> 4) if I end up wiping my device, I can just remove the bcache layer altogether.
> 
> > Even if a single drive doesn't reorder writes, multiple drives in
> > raid effectively reorder writes between drives as each drive has its
> > own distinct write queue.  A dropped write that would be harmless in a
> 
> very good point.
> 
> > single-device filesystem could be harmful in a multi-device array as the
> > non-failing drives will have writes that occurred after the lost writes
> > on the failing drive.  Normally mdadm enforces flush ordering across all
> > component devices so that this isn't a problem, but if you have optimistic
> > firmware and a drive failure after the flush command returns success,
> > the inter-drive ordering enforcement fails and the result is the same
> > as if an individual drive had a write/flush reordering bug.
> 
> That's all fair, but it feels like FS kept writing way longer than it
> was supposed to and that is what worries me the most.

The log excerpt I saw didn't show any write errors from btrfs, only
from mdadm.  That suggests to me that the failure happened earlier when
it was already too late to respond.  Also we can't rule out that bcache
is doing something stupid.

> > Some years ago we did a fleetwide audit of bad firmware drives and
> > found about a third of our drives were bad.  We disabled write cache on
> 
> Looks like I just want to disable write caching then. Correctness beats
> speed for sure.
> 
> > Yeah, experimental recovery code is fun, but robust backups and a working
> > disaster recovery plan is usually better.  Even if the filesystem is
> > up and running again, I'd want to compare all the files against backups
> > because I'd trust nothing on a filesystem after fsck touched it.
>  
> That is totally my plan. Given the output that I'm seeing, I'll definitely do
> at least a diff of all files between the backup and the array being recovered.
> I might even do an rsync, forcing md5 checksums, but if I do that, it
> will take days and longer than restoring the backup.
> 
> > On the other hand, if you get lucky and the filesystem isn't too badly
> > damaged, then comparing the data with backups will be more convenient
> > than starting over with a new filesystem.
> 
> Agreed.
> 
> Thanks,
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>  
> Home page: http://marc.merlins.org/  
> 

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06 19:16                                                                                                                                                                 ` Marc MERLIN
@ 2022-04-06 19:53                                                                                                                                                                   ` Josef Bacik
  2022-04-06 20:34                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-06 19:53 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Apr 6, 2022 at 3:16 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Apr 06, 2022 at 02:57:03PM -0400, Josef Bacik wrote:
> > Yeah lets go for that, I saw some errors on your fs tree's earlier, I
> > may need to adapt this to fix that tree, or if it's a snapshot we can
> > just delete it.  We can burn that bridge when we get to it, thanks,
>
> Sounds good. Is mode=lowmem a thing of the past by the way?
> (I remember that the old regular repair would eat up my 32GB of RAM and
> take the machine down)
>
> The current one failed quickly though
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a
> enabling repair mode
> WARNING:
>
>         Do not use --repair unless you are advised to do so by a developer
>         or an experienced user, and then only after having accepted that no
>         fsck can successfully repair all types of filesystem corruption. Eg.
>         some software or hardware bugs can fatally damage a volume.
>         The operation will start in 10 seconds.
>         Use Ctrl-C to stop it.
> 10 9 8 7 6 5 4 3 2 1
> Starting repair.
> Opening filesystem to check...
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> parent transid verify failed on 22216704 wanted 1600938 found 1602177
> FS_INFO IS 0x55e2e75defc0
> parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
> parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
> parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
> parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
> parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
> parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
> parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
> parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
> Couldn't find the last root for 8
> checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
> parent transid verify failed on 15645261971456 wanted 1602297 found 1600989
> checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
> bad tree block 15645261971456, bad level, 127 > 8
> ERROR: failed to read block groups: Input/output error
> FS_INFO AFTER IS 0x55e2e75defc0
> Checking filesystem on /dev/mapper/dshelf1a
> UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
> [1/7] checking root items
> checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
> parent transid verify failed on 15645261971456 wanted 1602297 found 1600989
> checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
> bad tree block 15645261971456, bad level, 127 > 8
> ERROR: failed to repair root items: Input/output error
>

Alright so it's up to you, clearly my put the tree back together stuff
takes forever, you can use --init-extent-tree with --lowmem if you'd
like, I have no idea what the time on that is going to look like.  You
still may run into problems with that if your subvols are screwed, but
then I just have to do the work to put subvols back together.  I
*think* the --init-extent-tree thing will be faster, but let me know
what you want to do.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06 19:53                                                                                                                                                                   ` Josef Bacik
@ 2022-04-06 20:34                                                                                                                                                                     ` Marc MERLIN
  2022-04-06 20:38                                                                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-06 20:34 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Apr 06, 2022 at 03:53:34PM -0400, Josef Bacik wrote:
> Alright so it's up to you, clearly my put the tree back together stuff
> takes forever, you can use --init-extent-tree with --lowmem if you'd
> like, I have no idea what the time on that is going to look like.  You
> still may run into problems with that if your subvols are screwed, but
> then I just have to do the work to put subvols back together.  I
> *think* the --init-extent-tree thing will be faster, but let me know
> what you want to do.  Thanks,

Thanks for sticking with me all this time.
So, it was pretty quick before it failed:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree --repair /dev/mapper/dshelf1a 
enabling repair mode
WARNING:

	Do not use --repair unless you are advised to do so by a developer
	or an experienced user, and then only after having accepted that no
	fsck can successfully repair all types of filesystem corruption. Eg.
	some software or hardware bugs can fatally damage a volume.
	The operation will start in 10 seconds.
	Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x556ddb2fffc0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x556ddb2fffc0
Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Creating a new extent tree
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6
checksum verify failed on 15645099524096 wanted 0x84d38588 found 0xebe9c53b
checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d
checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188
checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858
checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82
checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed
checksum verify failed on 15645256777728 wanted 0x6f64534d found 0x8547f2a5
checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c
checksum verify failed on 15645277880320 wanted 0x94805cc2 found 0xb68a1cef
checksum verify failed on 15645849485312 wanted 0x7afc435f found 0x6977ed7c
checksum verify failed on 15645146660864 wanted 0xfa138a06 found 0x10cddfd6
checksum verify failed on 15645610672128 wanted 0x2ceeada6 found 0x2db846f5
checksum verify failed on 15645208985600 wanted 0xbaaa42be found 0x769e23f0
checksum verify failed on 11822142046208 wanted 0xc4cc3d0a found 0xace45394
checksum verify failed on 11822142177280 wanted 0x1ac605a6 found 0x17c9826e
checksum verify failed on 6294487416832 wanted 0xe2be154a found 0xff9fb6b0
checksum verify failed on 11971034955776 wanted 0xcab8f347 found 0xdc99be9f
checksum verify failed on 15646027497472 wanted 0xfe85f37e found 0x4e0364ef
checksum verify failed on 15645802315776 wanted 0xdb3aba28 found 0x747ed64e
checksum verify failed on 15645883285504 wanted 0x62bd1000 found 0x7f1300cc
checksum verify failed on 13577053175808 wanted 0x4cec1ad0 found 0xf847830e
checksum verify failed on 15646038179840 wanted 0xe040a9e4 found 0x31183a57
checksum verify failed on 15646040834048 wanted 0x39f56fc2 found 0x336aee8b
checksum verify failed on 15646040604672 wanted 0xa2ff0db7 found 0xff9b0719
checksum verify failed on 15645608050688 wanted 0xe1019329 found 0x835ef4b7
checksum verify failed on 15646038212608 wanted 0x06fb70d5 found 0x704233fa
checksum verify failed on 15645668425728 wanted 0x12b8b29d found 0x7e5aee23
checksum verify failed on 15645461397504 wanted 0xf186173b found 0xd3930df4
checksum verify failed on 15645684269056 wanted 0x6222cf99 found 0x50b7b0dd
checksum verify failed on 15645870178304 wanted 0xf8950821 found 0x1743580d
checksum verify failed on 15646019764224 wanted 0xd96815db found 0xa62eb220
checksum verify failed on 15645870358528 wanted 0x722138e3 found 0x9dbf8691
checksum verify failed on 15646016913408 wanted 0xfe1e83f4 found 0x664795e4
checksum verify failed on 10194600787968 wanted 0x37da9d7c found 0xf7d76f11
checksum verify failed on 15645949083648 wanted 0x14efeeac found 0x46344821
checksum verify failed on 13577202368512 wanted 0x4f5909be found 0x91649194
checksum verify failed on 15645136207872 wanted 0x4a16db57 found 0x042787a6
checksum verify failed on 13577220849664 wanted 0x31a222f0 found 0x98e6f9d5
checksum verify failed on 13577388523520 wanted 0x254b7048 found 0x90c5f781
checksum verify failed on 15645210001408 wanted 0xc64a49c0 found 0x8959550a
checksum verify failed on 13577220997120 wanted 0xc1cf4d40 found 0xb20065f3
checksum verify failed on 15645422108672 wanted 0x63b9fb01 found 0xef400164
checksum verify failed on 365093765120 wanted 0xef058d7b found 0x6d2e4388
checksum verify failed on 15645943742464 wanted 0xa84c7cdb found 0x7435ced6
checksum verify failed on 15645944037376 wanted 0x1ce3f544 found 0x4110ee5a
checksum verify failed on 15645943545856 wanted 0xb7391689 found 0x57835854
checksum verify failed on 15645943644160 wanted 0xd112805c found 0x65133906
checksum verify failed on 15645220012032 wanted 0xb4cb7eb3 found 0xe7134503
checksum verify failed on 11971032399872 wanted 0x1b1c1591 found 0x62f69873
checksum verify failed on 15645123133440 wanted 0xe29086d7 found 0x2bb17c15
checksum verify failed on 15645117890560 wanted 0xf94e5973 found 0xdf7eb70e
checksum verify failed on 15645445554176 wanted 0x9dae4aed found 0x1087f009
checksum verify failed on 15645133553664 wanted 0x5840e80b found 0xa0492d9e
checksum verify failed on 15645125804032 wanted 0xa71f19d7 found 0xd0d29066
checksum verify failed on 15645870292992 wanted 0x764a5b63 found 0x65f2d096
checksum verify failed on 15645209886720 wanted 0xdc6c9ccf found 0xfa3ec3d2
checksum verify failed on 15645167927296 wanted 0xa4d01cb6 found 0x44fb24c2
checksum verify failed on 15645786374144 wanted 0xb7242739 found 0x969cecbb
checksum verify failed on 11971140157440 wanted 0x89f0ba19 found 0x761f18b3
checksum verify failed on 15645076209664 wanted 0xb5e74fba found 0x18d2d38b
checksum verify failed on 15645214834688 wanted 0x81dcde00 found 0xc22cbcc7
checksum verify failed on 15645786636288 wanted 0x1edd6683 found 0xeeafe616
checksum verify failed on 15645217849344 wanted 0xf7574b63 found 0x00ba9085
checksum verify failed on 15645652893696 wanted 0x0fbc611c found 0x9539e808
checksum verify failed on 15645734240256 wanted 0x6ab0998e found 0xb51c5e0c
checksum verify failed on 15645733978112 wanted 0xff835175 found 0x8f517448
checksum verify failed on 15645734010880 wanted 0x5363464a found 0x2aec1941
checksum verify failed on 13577168486400 wanted 0x56b26cca found 0xeb90f5da
checksum verify failed on 13577616375808 wanted 0x190c00d0 found 0x18c748c3
checksum verify failed on 13577184264192 wanted 0x44d0fbb6 found 0x270072ce
checksum verify failed on 13577226272768 wanted 0x14900bf7 found 0x2d0d3658
checksum verify failed on 13577199779840 wanted 0x39a4976f found 0xec18a40c
checksum verify failed on 13577199730688 wanted 0x3e524f64 found 0xe82c5965
checksum verify failed on 13577225994240 wanted 0xc011d76f found 0x67bc3401
checksum verify failed on 15645260890112 wanted 0x0a7005a3 found 0xd182e084
checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507
checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507
bad tree block 15645260890112, bad level, 59 > 8
Error reading tree block
error pinning down used bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13577814573056 len 16384

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06 20:34                                                                                                                                                                     ` Marc MERLIN
@ 2022-04-06 20:38                                                                                                                                                                       ` Josef Bacik
  2022-04-06 20:56                                                                                                                                                                         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-06 20:38 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Apr 6, 2022 at 4:34 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Apr 06, 2022 at 03:53:34PM -0400, Josef Bacik wrote:
> > Alright so it's up to you, clearly my put the tree back together stuff
> > takes forever, you can use --init-extent-tree with --lowmem if you'd
> > like, I have no idea what the time on that is going to look like.  You
> > still may run into problems with that if your subvols are screwed, but
> > then I just have to do the work to put subvols back together.  I
> > *think* the --init-extent-tree thing will be faster, but let me know
> > what you want to do.  Thanks,
>
> Thanks for sticking with me all this time.
> So, it was pretty quick before it failed:
>

Ah yeah I should have expected that, can you pull and re-run the same
command?  It'll tell me which root we need to target next.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: figuring out why transient double raid failure caused a fair amount of btrfs corruption
  2022-04-06 19:13                                                                                                                                       ` Marc MERLIN
  2022-04-06 19:45                                                                                                                                         ` Zygo Blaxell
@ 2022-04-06 20:38                                                                                                                                         ` Marc MERLIN
  2022-04-06 20:51                                                                                                                                           ` Josef Bacik
  2022-04-22 18:48                                                                                                                                         ` Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) Marc MERLIN
  2 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-06 20:38 UTC (permalink / raw)
  To: Zygo Blaxell; +Cc: Josef Bacik, linux-btrfs

This is an interesting discussion, so let's make a new thread out of it.
TL;DR: I think btrfs may have failed to go read only earlier, causing
more damage than needed to be, or some block layers just held enough
data in flight that partial data got written, causing more damage than
expected.
Figuring out the underlying problem would be good to avoid this again.

On Wed, Apr 06, 2022 at 02:07:47PM -0400, Zygo Blaxell wrote:
> Readahead can fail for a number of reasons that aren't real problems,

thanks.

> Once a degraded array has an IO failure, mdadm assumes you're in
> a data-scraping recovery mode and simply passes errors through.
> bcache does something similar when the backing store fails.
 
shouldn't it go read only also?
I haven't found a setting to tell it to do that if it's not the default.

> btrfs is the agent that has to stop attempting writes, and make sure
> any transaction in progress doesn't get committed.  ext4 has similar
> responsibility and implements its own force-read-only feature.

Agreed, but I like the defense in multiple layers approach
mdadm knows that any data going to be written is going to be incomplete
due to the 2nd missing drive, and there are few to no scenarios where 
continuing to write is a good thing.

> I note that btrfs is reporting only read errors here.

Yeah, me too. That seems very bad and likely the cause or the corruption
I got.
A double raid failure (I've had them before) should be almost a non
event. The first write after that should fail and that should be the end
of it.
This underlying problem makes me very uncomfortable.
As much as I'm super thankful for Josef's efforts, and ultimately it
will make btrfs recovery better, which is a win for all, my situation
shouldn't have happened from what I can tell.

> There's a possibility that the drive dropped the writes during the
> bus reset at the start of the second drive failure.  If write caching
> was enabled in the drive, and the drive has a firmware bug that drops
> the write cache contents (or it's just failing hardware, i.e. the CPU
> running the drive firmware is getting killed by electrical problems on the
> controller board, causing both the bus drop and the loss of write cache
> contents), then writes in the drive's cache could be lost _after_ mdadm,
> bcache and btrfs had been told by the drive that they were completed.

That's true, but I've seen btfrs remount read only before, and it didn't
there. Shouldn't hard IO errors immediately cause btrfs to go read only?

Write caching is on all those drives though:
gargamel:/var/local/src/btrfs-progs-josefbacik# hdparm -v -W /dev/sdh
/dev/sdh:
 multcount     = 16 (on)
 readonly      =  0 (off)
 readahead     = 256 (on)
 geometry      = 729601/255/63, sectors = 11721045168, start = 0
 write-caching =  1 (on)

I haven't heard that these drives have broken caching, but maybe they do?
Device Model:     ST6000VN0041-2EL11C
Serial Number:    ZA18TVFZ
LU WWN Device Id: 5 000c50 0a4d9b49c
Firmware Version: SC61
User Capacity:    6,001,175,126,016 bytes [6.00 TB]

> If the failing drive also reorders cached writes across flush
> commands, then we go directly to parent transid verify failed in btrfs.
> Parent transid verification is designed to detect this exact firmware
> failure mode, and it usually works as intended.  It's a pretty direct
> and reliable signal that write ordering is broken in a lower level of
> the storage stack, and must be fixed or disabled before trying again.
 
Agreed, I do want to fix the underlying problem here.
What I can do is

1) disable write caching on the drives

2) disable bcache by removing the caching device

3) change the cache mode
gargamel:/sys/block/md7/bcache# cat cache_mode 
[writethrough] writeback writearound none
writethrough should be safe, but I could use writearound instead

4) if I end up wiping my device, I can just remove the bcache layer altogether.

> Even if a single drive doesn't reorder writes, multiple drives in
> raid effectively reorder writes between drives as each drive has its
> own distinct write queue.  A dropped write that would be harmless in a

very good point.

> single-device filesystem could be harmful in a multi-device array as the
> non-failing drives will have writes that occurred after the lost writes
> on the failing drive.  Normally mdadm enforces flush ordering across all
> component devices so that this isn't a problem, but if you have optimistic
> firmware and a drive failure after the flush command returns success,
> the inter-drive ordering enforcement fails and the result is the same
> as if an individual drive had a write/flush reordering bug.

That's all fair, but it feels like FS kept writing way longer than it
was supposed to and that is what worries me the most.

> Some years ago we did a fleetwide audit of bad firmware drives and
> found about a third of our drives were bad.  We disabled write cache on

Looks like I just want to disable write caching then. Correctness beats
speed for sure.

> Yeah, experimental recovery code is fun, but robust backups and a working
> disaster recovery plan is usually better.  Even if the filesystem is
> up and running again, I'd want to compare all the files against backups
> because I'd trust nothing on a filesystem after fsck touched it.
 
That is totally my plan. Given the output that I'm seeing, I'll definitely do
at least a diff of all files between the backup and the array being recovered.
I might even do an rsync, forcing md5 checksums, but if I do that, it
will take days and longer than restoring the backup.

> On the other hand, if you get lucky and the filesystem isn't too badly
> damaged, then comparing the data with backups will be more convenient
> than starting over with a new filesystem.

Agreed.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: figuring out why transient double raid failure caused a fair amount of btrfs corruption
  2022-04-06 20:38                                                                                                                                         ` figuring out why transient double raid failure caused a fair amount of btrfs corruption Marc MERLIN
@ 2022-04-06 20:51                                                                                                                                           ` Josef Bacik
  2022-04-06 21:14                                                                                                                                             ` Marc MERLIN
  2022-04-07 12:27                                                                                                                                             ` Zygo Blaxell
  0 siblings, 2 replies; 479+ messages in thread
From: Josef Bacik @ 2022-04-06 20:51 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Zygo Blaxell, linux-btrfs

On Wed, Apr 6, 2022 at 4:38 PM Marc MERLIN <marc@merlins.org> wrote:
>
> This is an interesting discussion, so let's make a new thread out of it.
> TL;DR: I think btrfs may have failed to go read only earlier, causing
> more damage than needed to be, or some block layers just held enough
> data in flight that partial data got written, causing more damage than
> expected.
> Figuring out the underlying problem would be good to avoid this again.
>

There's a lot here, and there's a lot of good information, but in
general btrfs should never fail like this.  We have a lot of power
fail testing and error handling testing, but this is very limited.  I
found a bug last year where we would write an updated super block that
pointed to garbage if we happened to be fsync'ing at the same time we
were committing the transaction and got an error during the
transaction commit phase.

We can't do anything about the disks lying to us.  If a disk has a
wonky FUA/FLUSH implementation then we're just sort of screwed.
Unfortunately because our metadata moves around a lot we're waaaaay
more susceptible to this failure case than ext4 or xfs, their metadata
is relatively static they can put humpty dumpty back together again
relatively simply.

Btrfs needs to

1. Go whole hog on error injection testing.  I only barely scratched
the surface with my bpf error injection stuff.  This is on our roadmap
and I plan on devoting developer time to this, but clearly that
doesn't help you right now.
2. Put a lot more effort into disaster recovery.  What I've written
for you is an idea I've had in my head for a while.  Some of this
failures aren't catastrophic, we can generally pretty easily put back
together a file system that resembles something sane by simply
stitching together blocks that we find that are close enough to what
we wanted.  Unfortunately this gets back-burnered because in reality
this doesn't happen that often.
3. Test these btrfs+dmcrypt+mdraid setups.  Every time I notice one of
these catastrophic failures it generally involves btrfs+<something
else>.  This is likely just because it's a timing thing, you put more
layers you get a wider window in per-io races, you're more likely to
be sad in the event of a failure.  However it would be good to make
sure these layers are doing the correct thing themselves.

We need to be better about this scenario, both in making sure we don't
have bugs that contribute to the problem, but also that we have the
tools necessary to recover when things go wrong.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06 20:38                                                                                                                                                                       ` Josef Bacik
@ 2022-04-06 20:56                                                                                                                                                                         ` Marc MERLIN
  2022-04-06 21:05                                                                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-06 20:56 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Apr 06, 2022 at 04:38:06PM -0400, Josef Bacik wrote:
> Ah yeah I should have expected that, can you pull and re-run the same
> command?  It'll tell me which root we need to target next.  Thanks,

Not sure it helped.

gargamel:/var/local/src/btrfs-progs-josefbacik# l btrfs
-rwxr-xr-x 1 root staff 6217688 Apr  6 13:38 btrfs*
gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -5
commit d51753abbc0ce345b9e163fe0fdbff086c9c609d
Author: Josef Bacik <josef@toxicpanda.com>
Date:   Wed Apr 6 16:37:17 2022 -0400

    i need the owner

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree --repair /dev/mapper/dshelf1a 
enabling repair mode
WARNING:

	Do not use --repair unless you are advised to do so by a developer
	or an experienced user, and then only after having accepted that no
	fsck can successfully repair all types of filesystem corruption. Eg.
	some software or hardware bugs can fatally damage a volume.
	The operation will start in 10 seconds.
	Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x55b85082dfc0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x55b85082dfc0
Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Creating a new extent tree
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6
checksum verify failed on 15645099524096 wanted 0x84d38588 found 0xebe9c53b
checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d
checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188
checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858
checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82
checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed
checksum verify failed on 15645256777728 wanted 0x6f64534d found 0x8547f2a5
checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c
checksum verify failed on 15645277880320 wanted 0x94805cc2 found 0xb68a1cef
checksum verify failed on 15645849485312 wanted 0x7afc435f found 0x6977ed7c
checksum verify failed on 15645146660864 wanted 0xfa138a06 found 0x10cddfd6
checksum verify failed on 15645610672128 wanted 0x2ceeada6 found 0x2db846f5
checksum verify failed on 15645208985600 wanted 0xbaaa42be found 0x769e23f0
checksum verify failed on 11822142046208 wanted 0xc4cc3d0a found 0xace45394
checksum verify failed on 11822142177280 wanted 0x1ac605a6 found 0x17c9826e
checksum verify failed on 6294487416832 wanted 0xe2be154a found 0xff9fb6b0
checksum verify failed on 11971034955776 wanted 0xcab8f347 found 0xdc99be9f
checksum verify failed on 15646027497472 wanted 0xfe85f37e found 0x4e0364ef
checksum verify failed on 15645802315776 wanted 0xdb3aba28 found 0x747ed64e
checksum verify failed on 15645883285504 wanted 0x62bd1000 found 0x7f1300cc
checksum verify failed on 13577053175808 wanted 0x4cec1ad0 found 0xf847830e
checksum verify failed on 15646038179840 wanted 0xe040a9e4 found 0x31183a57
checksum verify failed on 15646040834048 wanted 0x39f56fc2 found 0x336aee8b
checksum verify failed on 15646040604672 wanted 0xa2ff0db7 found 0xff9b0719
checksum verify failed on 15645608050688 wanted 0xe1019329 found 0x835ef4b7
checksum verify failed on 15646038212608 wanted 0x06fb70d5 found 0x704233fa
checksum verify failed on 15645668425728 wanted 0x12b8b29d found 0x7e5aee23
checksum verify failed on 15645461397504 wanted 0xf186173b found 0xd3930df4
checksum verify failed on 15645684269056 wanted 0x6222cf99 found 0x50b7b0dd
checksum verify failed on 15645870178304 wanted 0xf8950821 found 0x1743580d
checksum verify failed on 15646019764224 wanted 0xd96815db found 0xa62eb220
checksum verify failed on 15645870358528 wanted 0x722138e3 found 0x9dbf8691
checksum verify failed on 15646016913408 wanted 0xfe1e83f4 found 0x664795e4
checksum verify failed on 10194600787968 wanted 0x37da9d7c found 0xf7d76f11
checksum verify failed on 15645949083648 wanted 0x14efeeac found 0x46344821
checksum verify failed on 13577202368512 wanted 0x4f5909be found 0x91649194
checksum verify failed on 15645136207872 wanted 0x4a16db57 found 0x042787a6
checksum verify failed on 13577220849664 wanted 0x31a222f0 found 0x98e6f9d5
checksum verify failed on 13577388523520 wanted 0x254b7048 found 0x90c5f781
checksum verify failed on 15645210001408 wanted 0xc64a49c0 found 0x8959550a
checksum verify failed on 13577220997120 wanted 0xc1cf4d40 found 0xb20065f3
checksum verify failed on 15645422108672 wanted 0x63b9fb01 found 0xef400164
checksum verify failed on 365093765120 wanted 0xef058d7b found 0x6d2e4388
checksum verify failed on 15645943742464 wanted 0xa84c7cdb found 0x7435ced6
checksum verify failed on 15645944037376 wanted 0x1ce3f544 found 0x4110ee5a
checksum verify failed on 15645943545856 wanted 0xb7391689 found 0x57835854
checksum verify failed on 15645943644160 wanted 0xd112805c found 0x65133906
checksum verify failed on 15645220012032 wanted 0xb4cb7eb3 found 0xe7134503
checksum verify failed on 11971032399872 wanted 0x1b1c1591 found 0x62f69873
checksum verify failed on 15645123133440 wanted 0xe29086d7 found 0x2bb17c15
checksum verify failed on 15645117890560 wanted 0xf94e5973 found 0xdf7eb70e
checksum verify failed on 15645445554176 wanted 0x9dae4aed found 0x1087f009
checksum verify failed on 15645133553664 wanted 0x5840e80b found 0xa0492d9e
checksum verify failed on 15645125804032 wanted 0xa71f19d7 found 0xd0d29066
checksum verify failed on 15645870292992 wanted 0x764a5b63 found 0x65f2d096
checksum verify failed on 15645209886720 wanted 0xdc6c9ccf found 0xfa3ec3d2
checksum verify failed on 15645167927296 wanted 0xa4d01cb6 found 0x44fb24c2
checksum verify failed on 15645786374144 wanted 0xb7242739 found 0x969cecbb
checksum verify failed on 11971140157440 wanted 0x89f0ba19 found 0x761f18b3
checksum verify failed on 15645076209664 wanted 0xb5e74fba found 0x18d2d38b
checksum verify failed on 15645214834688 wanted 0x81dcde00 found 0xc22cbcc7
checksum verify failed on 15645786636288 wanted 0x1edd6683 found 0xeeafe616
checksum verify failed on 15645217849344 wanted 0xf7574b63 found 0x00ba9085
checksum verify failed on 15645652893696 wanted 0x0fbc611c found 0x9539e808
checksum verify failed on 15645734240256 wanted 0x6ab0998e found 0xb51c5e0c
checksum verify failed on 15645733978112 wanted 0xff835175 found 0x8f517448
checksum verify failed on 15645734010880 wanted 0x5363464a found 0x2aec1941
checksum verify failed on 13577168486400 wanted 0x56b26cca found 0xeb90f5da
checksum verify failed on 13577616375808 wanted 0x190c00d0 found 0x18c748c3
checksum verify failed on 13577184264192 wanted 0x44d0fbb6 found 0x270072ce
checksum verify failed on 13577226272768 wanted 0x14900bf7 found 0x2d0d3658
checksum verify failed on 13577199779840 wanted 0x39a4976f found 0xec18a40c
checksum verify failed on 13577199730688 wanted 0x3e524f64 found 0xe82c5965
checksum verify failed on 13577225994240 wanted 0xc011d76f found 0x67bc3401
checksum verify failed on 15645260890112 wanted 0x0a7005a3 found 0xd182e084
checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507
checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507
bad tree block 15645260890112, bad level, 59 > 8
Error reading tree block owner 7
error pinning down used bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13577814573056 len 16384


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06 20:56                                                                                                                                                                         ` Marc MERLIN
@ 2022-04-06 21:05                                                                                                                                                                           ` Josef Bacik
  2022-04-07  1:08                                                                                                                                                                             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-06 21:05 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Apr 6, 2022 at 4:56 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Apr 06, 2022 at 04:38:06PM -0400, Josef Bacik wrote:
> > Ah yeah I should have expected that, can you pull and re-run the same
> > command?  It'll tell me which root we need to target next.  Thanks,
>
> Not sure it helped.
>

Yup that's what I needed, the csum tree is screwed, and actually the
reinit stuff won't ignore the csum root if we're also init'ing the
csum root which is kind of annoying.  I updated the btrfs-find-root
tool to do the tree repair thing for the csum root, you can run
./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you
can re-try the btrfs check.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: figuring out why transient double raid failure caused a fair amount of btrfs corruption
  2022-04-06 20:51                                                                                                                                           ` Josef Bacik
@ 2022-04-06 21:14                                                                                                                                             ` Marc MERLIN
  2022-04-07 12:27                                                                                                                                             ` Zygo Blaxell
  1 sibling, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-04-06 21:14 UTC (permalink / raw)
  To: Josef Bacik, Zygo Blaxell; +Cc: linux-btrfs

(Zygo, answer for you below)

On Wed, Apr 06, 2022 at 04:51:14PM -0400, Josef Bacik wrote:
> We can't do anything about the disks lying to us.  If a disk has a
> wonky FUA/FLUSH implementation then we're just sort of screwed.
> Unfortunately because our metadata moves around a lot we're waaaaay
> more susceptible to this failure case than ext4 or xfs, their metadata
> is relatively static they can put humpty dumpty back together again
> relatively simply.
 
Yeah, that's clearly been my experience. The average end user (which
kind of includes me), totally understands that last data written,
includig potentially directories, will get damaged, potentially beyond
repair, but losing the entire filesystem when 99.99%+ never got touched
by the last operations, is unexpected and undesired.
That said, yes, I understand that btrfs does things that are both cool
and complex, and they are less tolerant of corruption.

> 2. Put a lot more effort into disaster recovery.  What I've written
> for you is an idea I've had in my head for a while.  Some of this

I truly appreciate the time you've been spending with me on this.
In the end, I do have a backup, but I could not have one, or it could
fail when I'm reading back from it, so being able to recover the
filesystem to some workable state, is definitely a big plus.
Honestly I don't even really care if I lose 0.1%, 1%, or 5% of the data
(even if probably only <0.001% of the data got touched in the minute+
that btrfs was still apparently trying to use the filesystem after the
double failure), but a recovery that takes at most hours (for 24TB) and
not days, or never (never is a multiple day restore over the network for
me), will definitely be a great improvement.

> 3. Test these btrfs+dmcrypt+mdraid setups.  Every time I notice one of
> these catastrophic failures it generally involves btrfs+<something
> else>.  This is likely just because it's a timing thing, you put more
> layers you get a wider window in per-io races, you're more likely to
> be sad in the event of a failure.  However it would be good to make
> sure these layers are doing the correct thing themselves.
 
Totally agreed, I'm actually the worst case scenario, I have

disk hw write cache + swraid5 + bcache + dmcrypt + btrfs

that's clearly a lot of layers. This is now my 4 or 5th pretty
catastrophic btrfs failure over 2 arrays with those amount of layers
over the last 8 years, but the last one was 2-3 years ago, so there is
that :)
I think it's pretty clear that having all those layers is significantly
increasing my risk factor.

> We need to be better about this scenario, both in making sure we don't
> have bugs that contribute to the problem, but also that we have the
> tools necessary to recover when things go wrong.  Thanks,

agreed to both, thanks again for your time on this

On Wed, Apr 06, 2022 at 03:45:57PM -0400, Zygo Blaxell wrote:
> > shouldn't it go read only also?
> > I haven't found a setting to tell it to do that if it's not the default.
> 
> bcache in writethrough mode could leave cached blocks dirty as long as
> the SSD completes the write.  It should be reporting the write errors
> back to upper layers.
> Whether it actually does...I don't know, I haven't run that kind of
 
agreed.

> > That's true, but I've seen btfrs remount read only before, and it didn't
> > there. Shouldn't hard IO errors immediately cause btrfs to go read only?
> 
> No, only hard write IO errors on all metadata mirror drives, and cases
> where btrfs needs to CoW a page or read free space tree, and can't find
> an intact mirror.  Anything less is correctable (write failure on some
> mirrors) or can be retried (any read failure) if the raid profile has
> redundancy.

Fair enough. And as you said, my errors were probably already staged in
multiple layers and flushing before I got the first error reported back,
so even by then it mau have been too late to stop it.

> > I haven't heard that these drives have broken caching, but maybe they do?
> > Device Model:     ST6000VN0041-2EL11C
> > Serial Number:    ZA18TVFZ
> > LU WWN Device Id: 5 000c50 0a4d9b49c
> > Firmware Version: SC61
> > User Capacity:    6,001,175,126,016 bytes [6.00 TB]
> 
> I've heard of problems with SC60, but IIRC SC61 was supposed to be the

Well, there you go, just found out that I have a mix of SC60 and SC61.
Ok, I'll just turn off write caching on them all.

> The log excerpt I saw didn't show any write errors from btrfs, only
> from mdadm.  That suggests to me that the failure happened earlier when
> it was already too late to respond.  Also we can't rule out that bcache
> is doing something stupid.
 
Yeah, at this point I think the logs agree with that statement.
Also, I wasn't actually using that array, stuff doesn't really get
written to it unless I manually start copies.

Thanks for looking this over and your opinion/sanity check.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06 21:05                                                                                                                                                                           ` Josef Bacik
@ 2022-04-07  1:08                                                                                                                                                                             ` Marc MERLIN
  2022-04-07  1:18                                                                                                                                                                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-07  1:08 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Apr 06, 2022 at 05:05:25PM -0400, Josef Bacik wrote:
> Yup that's what I needed, the csum tree is screwed, and actually the
> reinit stuff won't ignore the csum root if we're also init'ing the
> csum root which is kind of annoying.  I updated the btrfs-find-root
> tool to do the tree repair thing for the csum root, you can run
> ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you
> can re-try the btrfs check.  Thanks,

Different output going back and forth, expected?

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x55ba30af45f0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x55ba30af45f0
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 62
fixed slot 88
fixed slot 235
fixed slot 237
Couldn't find a replacement block for slot 238
Couldn't find a replacement block for slot 243
fixed slot 33
fixed slot 35
fixed slot 32
fixed slot 133
fixed slot 136
fixed slot 207
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 484
Couldn't find a replacement block for slot 76
fixed slot 229
fixed slot 172
fixed slot 174
fixed slot 36
Couldn't find a replacement block for slot 20
fixed slot 24
Couldn't find a replacement block for slot 28
fixed slot 115
fixed slot 148
fixed slot 9
Couldn't find a replacement block for slot 318
fixed slot 245
fixed slot 1
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 134
fixed slot 159
fixed slot 170
fixed slot 204
fixed slot 149
fixed slot 185
fixed slot 257
fixed slot 46
fixed slot 60
fixed slot 91
fixed slot 92
fixed slot 95
fixed slot 247
fixed slot 105
Couldn't find a replacement block for slot 109
fixed slot 124
fixed slot 191
fixed slot 244
fixed slot 9
fixed slot 57
Couldn't find a replacement block for slot 166
fixed slot 163
fixed slot 78
fixed slot 179
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 218
fixed slot 26
fixed slot 75
Couldn't find a replacement block for slot 68
fixed slot 56
fixed slot 60
Couldn't find a replacement block for slot 231
fixed slot 141
fixed slot 151
fixed slot 156
fixed slot 159
fixed slot 163
fixed slot 171
fixed slot 200
fixed slot 5
Couldn't find a replacement block for slot 157
fixed slot 224
fixed slot 44
fixed slot 217
fixed slot 122
fixed slot 59
Couldn't find a replacement block for slot 120
fixed slot 181
fixed slot 187
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 220
fixed slot 60
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 192
fixed slot 9
fixed slot 7
Couldn't find a replacement block for slot 226
fixed slot 24
fixed slot 49
fixed slot 22
Couldn't find a replacement block for slot 186
fixed slot 149
fixed slot 150
Couldn't find a replacement block for slot 42
fixed slot 218
fixed slot 223
fixed slot 228
fixed slot 230
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 155
fixed slot 24
fixed slot 30
fixed slot 376
fixed slot 94
fixed slot 104
fixed slot 197
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 191
fixed slot 139
fixed slot 128
Couldn't find a replacement block for slot 143
fixed slot 16
fixed slot 17
fixed slot 114
fixed slot 28
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 83
fixed slot 86
fixed slot 92
fixed slot 149
fixed slot 117
Couldn't find a replacement block for slot 31
fixed slot 188
Couldn't find a replacement block for slot 102
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 79
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 21
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 276
Couldn't find a replacement block for slot 282
Couldn't find a replacement block for slot 283
Couldn't find a replacement block for slot 284
Couldn't find a replacement block for slot 285
Couldn't find a replacement block for slot 286
Couldn't find a replacement block for slot 287
Couldn't find a replacement block for slot 289
Couldn't find a replacement block for slot 290
Couldn't find a replacement block for slot 291
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 231
fixed slot 208
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 116

(still running)
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-07  1:08                                                                                                                                                                             ` Marc MERLIN
@ 2022-04-07  1:18                                                                                                                                                                               ` Josef Bacik
  2022-04-07  4:37                                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-07  1:18 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Apr 6, 2022 at 9:08 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Apr 06, 2022 at 05:05:25PM -0400, Josef Bacik wrote:
> > Yup that's what I needed, the csum tree is screwed, and actually the
> > reinit stuff won't ignore the csum root if we're also init'ing the
> > csum root which is kind of annoying.  I updated the btrfs-find-root
> > tool to do the tree repair thing for the csum root, you can run
> > ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you
> > can re-try the btrfs check.  Thanks,
>
> Different output going back and forth, expected?
>

Yup it's iterating over all the blocks, so it's going to find
different slots in different blocks that are fucked.

You are having a lot of ones we can't find good matches for, depending
on what those look like I may need to adjust the code to simply delete
slots we don't find a good match for, which will be fine since we're
going to clear this tree anyway.  But let it finish and we'll see how
the repair goes and I can do that if we need to.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-07  1:18                                                                                                                                                                               ` Josef Bacik
@ 2022-04-07  4:37                                                                                                                                                                                 ` Marc MERLIN
  2022-04-07  4:40                                                                                                                                                                                   ` Marc MERLIN
  2022-04-07  5:20                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 2 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-04-07  4:37 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Apr 06, 2022 at 09:18:17PM -0400, Josef Bacik wrote:
> On Wed, Apr 6, 2022 at 9:08 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Wed, Apr 06, 2022 at 05:05:25PM -0400, Josef Bacik wrote:
> > > Yup that's what I needed, the csum tree is screwed, and actually the
> > > reinit stuff won't ignore the csum root if we're also init'ing the
> > > csum root which is kind of annoying.  I updated the btrfs-find-root
> > > tool to do the tree repair thing for the csum root, you can run
> > > ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you
> > > can re-try the btrfs check.  Thanks,
> >
> > Different output going back and forth, expected?
> >
> 
> Yup it's iterating over all the blocks, so it's going to find
> different slots in different blocks that are fucked.
> 
> You are having a lot of ones we can't find good matches for, depending
> on what those look like I may need to adjust the code to simply delete
> slots we don't find a good match for, which will be fine since we're
> going to clear this tree anyway.  But let it finish and we'll see how
> the repair goes and I can do that if we need to.  Thanks,
> 
It took a lot longer, until it mostly hosed my machine by taking all the memory (32GB)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x55ba30af45f0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x55ba30af45f0
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 62
fixed slot 88
fixed slot 235
fixed slot 237
Couldn't find a replacement block for slot 238
Couldn't find a replacement block for slot 243
fixed slot 33
fixed slot 35
fixed slot 32
fixed slot 133
fixed slot 136
fixed slot 207
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 484
Couldn't find a replacement block for slot 76
fixed slot 229
fixed slot 172
fixed slot 174
fixed slot 36
Couldn't find a replacement block for slot 20
fixed slot 24
Couldn't find a replacement block for slot 28
fixed slot 115
fixed slot 148
fixed slot 9
Couldn't find a replacement block for slot 318
fixed slot 245
fixed slot 1
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 134
fixed slot 159
fixed slot 170
fixed slot 204
fixed slot 149
fixed slot 185
fixed slot 257
fixed slot 46
fixed slot 60
fixed slot 91
fixed slot 92
fixed slot 95
fixed slot 247
fixed slot 105
Couldn't find a replacement block for slot 109
fixed slot 124
fixed slot 191
fixed slot 244
fixed slot 9
fixed slot 57
Couldn't find a replacement block for slot 166
fixed slot 163
fixed slot 78
fixed slot 179
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 218
fixed slot 26
fixed slot 75
Couldn't find a replacement block for slot 68
fixed slot 56
fixed slot 60
Couldn't find a replacement block for slot 231
fixed slot 141
fixed slot 151
fixed slot 156
fixed slot 159
fixed slot 163
fixed slot 171
fixed slot 200
fixed slot 5
Couldn't find a replacement block for slot 157
fixed slot 224
fixed slot 44
fixed slot 217
fixed slot 122
fixed slot 59
Couldn't find a replacement block for slot 120
fixed slot 181
fixed slot 187
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 220
fixed slot 60
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 192
fixed slot 9
fixed slot 7
Couldn't find a replacement block for slot 226
fixed slot 24
fixed slot 49
fixed slot 22
Couldn't find a replacement block for slot 186
fixed slot 149
fixed slot 150
Couldn't find a replacement block for slot 42
fixed slot 218
fixed slot 223
fixed slot 228
fixed slot 230
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 155
fixed slot 24
fixed slot 30
fixed slot 376
fixed slot 94
fixed slot 104
fixed slot 197
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 191
fixed slot 139
fixed slot 128
Couldn't find a replacement block for slot 143
fixed slot 16
fixed slot 17
fixed slot 114
fixed slot 28
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 83
fixed slot 86
fixed slot 92
fixed slot 149
fixed slot 117
Couldn't find a replacement block for slot 31
fixed slot 188
Couldn't find a replacement block for slot 102
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 79
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 21
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 276
Couldn't find a replacement block for slot 282
Couldn't find a replacement block for slot 283
Couldn't find a replacement block for slot 284
Couldn't find a replacement block for slot 285
Couldn't find a replacement block for slot 286
Couldn't find a replacement block for slot 287
Couldn't find a replacement block for slot 289
Couldn't find a replacement block for slot 290
Couldn't find a replacement block for slot 291
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 231
fixed slot 208
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 208
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 130
fixed slot 138
fixed slot 33
Couldn't find a replacement block for slot 179
fixed slot 154
Couldn't find a replacement block for slot 256
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 129
fixed slot 111
Couldn't find a replacement block for slot 103
Couldn't find a replacement block for slot 192
fixed slot 12
Couldn't find a replacement block for slot 205
fixed slot 24
fixed slot 28
fixed slot 31
fixed slot 38
fixed slot 47
fixed slot 62
fixed slot 75
fixed slot 86
fixed slot 89
fixed slot 93
fixed slot 108
Couldn't find a replacement block for slot 115
fixed slot 146
fixed slot 218
fixed slot 288
fixed slot 33
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 176
fixed slot 240
Couldn't find a replacement block for slot 241
fixed slot 34
fixed slot 35
Couldn't find a replacement block for slot 71
Couldn't find a replacement block for slot 140
fixed slot 213
Couldn't find a replacement block for slot 155
fixed slot 210
fixed slot 111
fixed slot 147
fixed slot 150
Couldn't find a replacement block for slot 234
Couldn't find a replacement block for slot 236
fixed slot 240
fixed slot 250
Couldn't find a replacement block for slot 259
fixed slot 269
fixed slot 283
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 48
fixed slot 135
fixed slot 192
fixed slot 206
fixed slot 43
fixed slot 111
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 45
fixed slot 47
fixed slot 212
fixed slot 49
fixed slot 190
fixed slot 4
fixed slot 40
fixed slot 110
fixed slot 227
fixed slot 290
fixed slot 296
fixed slot 297
fixed slot 302
fixed slot 349
fixed slot 357
Couldn't find a replacement block for slot 53
fixed slot 304
Couldn't find a replacement block for slot 328
fixed slot 376
fixed slot 54
fixed slot 125
fixed slot 195
Couldn't find a replacement block for slot 89
fixed slot 177
fixed slot 197
Couldn't find a replacement block for slot 219
fixed slot 233
fixed slot 36
fixed slot 52
fixed slot 61
fixed slot 71
fixed slot 113
fixed slot 119
fixed slot 138
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 200
fixed slot 215
Couldn't find a replacement block for slot 173
fixed slot 315
fixed slot 320
fixed slot 326
fixed slot 328
fixed slot 335
fixed slot 65
fixed slot 32
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 34
Couldn't find a replacement block for slot 37
fixed slot 40
Couldn't find a replacement block for slot 179
fixed slot 75
fixed slot 198
fixed slot 79
fixed slot 16
fixed slot 89
fixed slot 111
fixed slot 152
fixed slot 219
fixed slot 220
fixed slot 43
fixed slot 70
fixed slot 85
fixed slot 102
fixed slot 130
fixed slot 145
fixed slot 146
fixed slot 161
fixed slot 188
fixed slot 202
Couldn't find a replacement block for slot 9
fixed slot 62
fixed slot 122
fixed slot 128
fixed slot 137
fixed slot 142
fixed slot 216
Couldn't find a replacement block for slot 26
fixed slot 143
fixed slot 152
fixed slot 174
fixed slot 185
fixed slot 76
fixed slot 87
fixed slot 26
fixed slot 28
fixed slot 31
fixed slot 33
fixed slot 35
fixed slot 43
fixed slot 46
fixed slot 47
fixed slot 56
fixed slot 62
fixed slot 69
fixed slot 73
fixed slot 76
fixed slot 78
fixed slot 81
fixed slot 83
fixed slot 90
fixed slot 92
fixed slot 93
fixed slot 94
fixed slot 98
fixed slot 110
fixed slot 113
fixed slot 114
fixed slot 115
fixed slot 117
fixed slot 118
fixed slot 119
fixed slot 120
fixed slot 121
fixed slot 122
fixed slot 125
fixed slot 142
fixed slot 146
fixed slot 147
fixed slot 148
fixed slot 179
fixed slot 182
fixed slot 185
fixed slot 101
Couldn't find a replacement block for slo^[[2~Timeout, server gargamel not responding.




[729314.382500] systemd-udevd[25274]: dm-13: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729314.490868] systemd-udevd[25258]: dm-12: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729351.745188] systemd-udevd[2257]: sda: Worker [25268] processing SEQNUM=101206 is taking a long time
[729352.745164] systemd-udevd[2257]: sdc: Worker [25273] processing SEQNUM=101228 is taking a long time
[729352.774312] systemd-udevd[2257]: wakeup25: Worker [25247] processing SEQNUM=101312 is taking a long time
[729354.745172] systemd-udevd[2257]: memory229: Worker [25275] processing SEQNUM=101499 is taking a long time
[729354.745174] systemd-udevd[25306]: memory230: Spawned process 'socket:/org/freedesktop/hal/udev_event' [26406] is taking longer than 59s to complete
[729354.745175] systemd-udevd[25275]: memory229: Spawned process 'socket:/org/freedesktop/hal/udev_event' [26401] is taking longer than 59s to complete
[729354.859212] systemd-udevd[2257]: memory238: Worker [25257] processing SEQNUM=101509 is taking a long time
[729354.889966] systemd-udevd[2257]: memory236: Worker [25246] processing SEQNUM=101507 is taking a long time
[729354.920686] systemd-udevd[2257]: memory234: Worker [25262] processing SEQNUM=101505 is taking a long time
[729354.951365] systemd-udevd[2257]: memory230: Worker [25306] processing SEQNUM=101501 is taking a long time
[729357.745162] systemd-udevd[2257]: md12: Worker [25269] processing SEQNUM=101854 is taking a long time
[729357.774644] systemd-udevd[2257]: sde: Worker [25263] processing SEQNUM=101860 is taking a long time
[729357.803818] systemd-udevd[2257]: md13: Worker [25340] processing SEQNUM=101855 is taking a long time
[729471.745176] systemd-udevd[2257]: sda: Worker [25268] processing SEQNUM=101206 killed
[729472.745177] systemd-udevd[2257]: sdc: Worker [25273] processing SEQNUM=101228 killed
[729472.769590] systemd-udevd[2257]: wakeup25: Worker [25247] processing SEQNUM=101312 killed
[729474.745171] systemd-udevd[25306]: memory230: Spawned process 'socket:/org/freedesktop/hal/udev_event' [26406] timed out after 2min 59s, killing
[729474.745175] systemd-udevd[2257]: memory229: Worker [25275] processing SEQNUM=101499 killed
[729474.812102] systemd-udevd[2257]: memory238: Worker [25257] processing SEQNUM=101509 killed
[729474.838576] systemd-udevd[2257]: memory230: Worker [25306] processing SEQNUM=101501 killed
[729474.864496] systemd-udevd[2257]: memory236: Worker [25246] processing SEQNUM=101507 killed
[729474.890429] systemd-udevd[2257]: memory234: Worker [25262] processing SEQNUM=101505 killed
[729474.916324] systemd-udevd[2257]: Worker [25275] terminated by signal 9 (KILL)
[729477.442170] systemd-udevd[25244]: sdd4: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729477.465202] systemd-udevd[25248]: sdd1: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729477.475194] systemd-udevd[2257]: sde: Worker [25263] processing SEQNUM=101860 killed
[729477.531960] systemd-udevd[2257]: md12: Worker [25269] processing SEQNUM=101854 killed
[729477.556548] systemd-udevd[2257]: md13: Worker [25340] processing SEQNUM=101855 killed
[729494.433414] systemd-udevd[25271]: sdd2: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729512.577884] systemd-udevd[25265]: sdd3: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729572.745171] systemd-udevd[2257]: sdd: Worker [25248] processing SEQNUM=101875 is taking a long time
[729692.745172] systemd-udevd[2257]: sdd: Worker [25248] processing SEQNUM=101875 killed
[729828.852640] systemd-udevd[25271]: LNXSYSTM:00: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.880567] systemd-udevd[25294]: cstate_core: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.882126] systemd-udevd[25265]: breakpoint: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.885879] systemd-udevd[25244]: cpu: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.885886] systemd-udevd[25339]: kprobe: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.886061] systemd-udevd[25252]: parport0: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.886527] systemd-udevd[25258]: cstate_pkg: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.888578] systemd-udevd[25255]: 0000:00:00.0: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.888587] systemd-udevd[25274]: i915: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.905903] systemd-udevd[25264]: msr: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729829.377862] elogind-daemon[9054]: Watching system buttons on /dev/input/event0 (Power Button)
[729829.469868] elogind-daemon[9054]: Watching system buttons on /dev/input/event1 (Power Button)
[729829.833865] elogind-daemon[9054]: Watching system buttons on /dev/input/event2 (AT Translated Set 2 keyboard)
[729830.359362] BTRFS info (device dm-12): devid 1 device path /dev/mapper/vgds2-1Appliances changed to /dev/dm-12 scanned by systemd-udevd (25274)
[729889.745171] systemd-udevd[2257]: 4-1.6.3: Worker [25264] processing SEQNUM=102334 is taking a long time
[730009.745168] systemd-udevd[2257]: 4-1.6.3: Worker [25264] processing SEQNUM=102334 killed
[730009.770590] systemd-udevd[2257]: Worker [25264] terminated by signal 9 (KILL)
[730009.793062] systemd-udevd[2257]: 4-1.6.3: Worker [25264] failed
[730009.813461] systemd-udevd[29139]: 4-1.6.3:1.0: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[732952.473804] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 35s!
[732952.498625] Showing busy workqueues and worker pools:
[732952.514833] workqueue events: flags=0x0
[732952.527430]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2
[732952.527442]     pending: cache_reap
[732952.527447]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[732952.527449]     pending: drm_fb_helper_damage_work [drm_kms_helper]
[732952.527472] workqueue writeback: flags=0x4a
[732952.615583]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4
[732952.615587]     in-flight: 8793:wb_workfn, 8613:wb_workfn
[732952.615608] workqueue btrfs-delalloc: flags=0xe
[732952.667841]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[732952.667843]     in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper
[732952.667859] workqueue btrfs-worker: flags=0xe
[732952.723431]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[732952.723433]     in-flight: 25134:btrfs_work_helper
[732952.723436] workqueue btrfs-endio-write: flags=0xe
[732952.774041]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[732952.774043]     in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper
[732952.774047] workqueue btrfs-delalloc: flags=0xe
[732952.830056]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[732952.830058]     in-flight: 25030:btrfs_work_helper
[732952.830070] workqueue cifsiod: flags=0xc
[732952.878011]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2
[732952.878013]     in-flight: 9189:cifs_echo_request [cifs]
[732952.878056] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=36s workers=2 manager: 17834
[732952.878059] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 idle: 2903 30021
[732983.193804] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 66s!
[732983.218550] Showing busy workqueues and worker pools:
[732983.234716] workqueue events: flags=0x0
[732983.247202]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2
[732983.247205]     pending: cache_reap
[732983.247209]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[732983.247211]     pending: drm_fb_helper_damage_work [drm_kms_helper]
[732983.247233] workqueue writeback: flags=0x4a
[732983.334945]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4
[732983.334948]     in-flight: 8793:wb_workfn, 8613:wb_workfn
[732983.334971] workqueue btrfs-delalloc: flags=0xe
[732983.386983]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[732983.386985]     in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper
[732983.387000] workqueue btrfs-worker: flags=0xe
[732983.442412]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[732983.442415]     in-flight: 25134:btrfs_work_helper
[732983.442417] workqueue btrfs-endio-write: flags=0xe
[732983.492919]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[732983.492921]     in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper
[732983.492925] workqueue btrfs-delalloc: flags=0xe
[732983.548832]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[732983.548843]     in-flight: 25030:btrfs_work_helper
[732983.548854] workqueue cifsiod: flags=0xc
[732983.596634]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2
[732983.596637]     in-flight: 9189:cifs_echo_request [cifs]
[732983.596680] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=66s workers=2 manager: 17834
[732983.596683] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 idle: 2903
[733013.909809] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 97s!

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-07  4:37                                                                                                                                                                                 ` Marc MERLIN
@ 2022-04-07  4:40                                                                                                                                                                                   ` Marc MERLIN
  2022-04-07  7:30                                                                                                                                                                                     ` Martin Steigerwald
  2022-04-07  5:20                                                                                                                                                                                   ` Marc MERLIN
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-07  4:40 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

Damn, it's hosed so badly that even sysrq-o isn't working, never seen
that before.

[733866.717217] sysrq: Terminate All Tasks
[733866.740424] elogind-daemon[9054]: Received signal 15 [TERM]
[733868.540292] sysrq: Emergency Remount R/O
[733870.249149] sysrq: Emergency Sync
[733904.793816] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 988s!
[733904.818717] Showing busy workqueues and worker pools:
[733904.834771] workqueue events: flags=0x0
[733904.847171]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=5/256 refcnt=6
[733904.847174]     pending: cache_reap, drm_fb_helper_damage_work [drm_kms_helper], free_work, do_emergency_remount, do_sync_work
[733904.847205] workqueue writeback: flags=0x4a
[733904.917846]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4
[733904.917849]     in-flight: 8793:wb_workfn, 8613:wb_workfn
[733904.917868] workqueue btrfs-delalloc: flags=0xe
[733904.969635]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[733904.969637]     in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper
[733904.969652] workqueue btrfs-worker: flags=0xe
[733905.024811]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[733905.024813]     in-flight: 25134:btrfs_work_helper
[733905.024817] workqueue btrfs-endio-write: flags=0xe
[733905.075033]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[733905.075035]     in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper
[733905.075039] workqueue btrfs-delalloc: flags=0xe
[733905.130695]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[733905.130697]     in-flight: 25030:btrfs_work_helper
[733905.130709] workqueue cifsiod: flags=0xc
[733905.178289]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=2/256 refcnt=4
[733905.178291]     in-flight: 9189:cifs_echo_request [cifs], 26186(RESCUER):cifs_resolve_server [cifs]
[733905.178363] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=988s workers=2 manager: 17834
[733905.178366] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 idle: 2903
[733909.916792] sysrq: Power Off
[733966.261816] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x1 nice=0 stuck for 55s!
[733966.286477] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 1049s!
[733966.311622] Showing busy workqueues and worker pools:
[733966.327664] workqueue events: flags=0x0
[733966.340063]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=5/256 refcnt=6
[733966.340067]     pending: cache_reap, drm_fb_helper_damage_work [drm_kms_helper], free_work, do_emergency_remount, do_sync_work
[733966.340093]   pwq 0: cpus=0 node=0 flags=0x1 nice=0 active=3/256 refcnt=4
[733966.340095]     in-flight: 17295:do_poweroff
[733966.340098]     pending: vmstat_shepherd, cache_reap
[733966.340106] workqueue writeback: flags=0x4a
[733966.461521]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4
[733966.461524]     in-flight: 8793:wb_workfn, 8613:wb_workfn
[733966.461543] workqueue btrfs-delalloc: flags=0xe
[733966.513325]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[733966.513328]     in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper
[733966.513344] workqueue btrfs-worker: flags=0xe
[733966.568487]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[733966.568489]     in-flight: 25134:btrfs_work_helper
[733966.568492] workqueue btrfs-endio-write: flags=0xe
[733966.618683]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[733966.618685]     in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper
[733966.618698] workqueue btrfs-delalloc: flags=0xe
[733966.674361]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[733966.674364]     in-flight: 25030:btrfs_work_helper
[733966.674376] workqueue cifsiod: flags=0xc
[733966.721929]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=2/256 refcnt=4
[733966.721932]     in-flight: 9189:cifs_echo_request [cifs], 26186(RESCUER):cifs_resolve_server [cifs]
[733966.722002] pool 0: cpus=0 node=0 flags=0x1 nice=0 hung=55s workers=2 manager: 9899
[733966.722006] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=1049s workers=2 manager: 17834
[733966.722008] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-07  4:37                                                                                                                                                                                 ` Marc MERLIN
  2022-04-07  4:40                                                                                                                                                                                   ` Marc MERLIN
@ 2022-04-07  5:20                                                                                                                                                                                   ` Marc MERLIN
  2022-04-07 16:29                                                                                                                                                                                     ` Marc MERLIN
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-07  5:20 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Apr 06, 2022 at 09:37:17PM -0700, Marc MERLIN wrote:
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo

there may have been something else that ate all my memory.
I killed shinobi and re-ran your code overnight while dumping
memory. I'll have a memory trace running this time while it re-runs
overnight.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-07  4:40                                                                                                                                                                                   ` Marc MERLIN
@ 2022-04-07  7:30                                                                                                                                                                                     ` Martin Steigerwald
  0 siblings, 0 replies; 479+ messages in thread
From: Martin Steigerwald @ 2022-04-07  7:30 UTC (permalink / raw)
  To: Josef Bacik, Marc MERLIN; +Cc: Linux BTRFS mailing list

Hi!

Marc MERLIN - 07.04.22, 06:40:06 CEST:
> Damn, it's hosed so badly that even sysrq-o isn't working, never seen
> that before.

I did not keep statistics, but this might be one of the longest threads 
on BTRFS mailing list.

Good luck with restoring your filesystem or at least recovering all data 
from it!

Best,
-- 
Martin



^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: figuring out why transient double raid failure caused a fair amount of btrfs corruption
  2022-04-06 20:51                                                                                                                                           ` Josef Bacik
  2022-04-06 21:14                                                                                                                                             ` Marc MERLIN
@ 2022-04-07 12:27                                                                                                                                             ` Zygo Blaxell
  1 sibling, 0 replies; 479+ messages in thread
From: Zygo Blaxell @ 2022-04-07 12:27 UTC (permalink / raw)
  To: Josef Bacik; +Cc: Marc MERLIN, linux-btrfs

On Wed, Apr 06, 2022 at 04:51:14PM -0400, Josef Bacik wrote:
> On Wed, Apr 6, 2022 at 4:38 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > This is an interesting discussion, so let's make a new thread out of it.
> > TL;DR: I think btrfs may have failed to go read only earlier, causing
> > more damage than needed to be, or some block layers just held enough
> > data in flight that partial data got written, causing more damage than
> > expected.
> > Figuring out the underlying problem would be good to avoid this again.

> We can't do anything about the disks lying to us.  If a disk has a
> wonky FUA/FLUSH implementation then we're just sort of screwed.
> Unfortunately because our metadata moves around a lot we're waaaaay
> more susceptible to this failure case than ext4 or xfs, their metadata
> is relatively static they can put humpty dumpty back together again
> relatively simply.

Disks are pretty good at FUA/FLUSH bugs these days, though you can
still buy new old drives that have them.

SSDs have a whole lot of bugs, especially if they get old. have a few bits
fail, and try to recover themselves, but blow up the whole filesystem
by losing a few pages.  These ones get through firmware qualification
testing because they only misbehave when they're a year or two old.

> Btrfs needs to
> 
> 1. Go whole hog on error injection testing.  I only barely scratched
> the surface with my bpf error injection stuff.  This is on our roadmap
> and I plan on devoting developer time to this, but clearly that
> doesn't help you right now.
> 2. Put a lot more effort into disaster recovery.  What I've written
> for you is an idea I've had in my head for a while.  Some of this
> failures aren't catastrophic, we can generally pretty easily put back
> together a file system that resembles something sane by simply
> stitching together blocks that we find that are close enough to what
> we wanted.  Unfortunately this gets back-burnered because in reality
> this doesn't happen that often.

This does seem to be an important missing piece at the moment.  
In practice, when we see "parent transid verify failed," we go directly
to mkfs + restore backups, as none of the existing tools will touch it.
They all want intact interior nodes in some tree to work, and that's
the one thing that FUA/FLUSH bugs destroy.

It doesn't solve the underlying problem--the drive will still trash
the filesystem every other month until its write ordering gets fixed or
worked around--but at least it lets users fix the filesystem in place
without mkfs + restoring a full set of backups.

> 3. Test these btrfs+dmcrypt+mdraid setups.  Every time I notice one of
> these catastrophic failures it generally involves btrfs+<something
> else>.  This is likely just because it's a timing thing, you put more
> layers you get a wider window in per-io races, you're more likely to
> be sad in the event of a failure.  However it would be good to make
> sure these layers are doing the correct thing themselves.

bcache and dmcache layers in series with the underlying storage multiply
the failure rates (including software regressions) of the individual
components, and mdadm blends firmware bugs from all the drives together.
Statistically we're always going to see more problems if there are more
moving parts in the system, especially if they aggregate risks and break
fault isolation.

FWIW I'm not seeing a difference in failure rate between "btrfs +
mdadm" and "btrfs alone", but I'm intentionally avoiding the many mdadm
configurations that introduce new failure modes by design (e.g. raid5
without ppl or journal, raid1 without component device integrity, or
single/raid1 SSD cache in writeback mode running over multiple drives).
There isn't a fix for those except to not use them.  In most cases
there's a better way to do the same thing, though there are some gaps
(e.g. there's no working solution for writeback SSD caching on raid5).

Thought experiment:  if you have 2 drives from vendor A and 2 drives
from vendor B, and you want to build two filesystems that replicate
to each other, where do you put the drives if you know one vendor has
a firmware bug but you don't know which one?  With mdadm, you build
filesystems with AA and BB drives, because that way a firmware bug
corrupts one filesystems and leaves the other intact.  With btrfs, you
build two filesystems with AB and AB drives, because that way a firmware
bug gets autocorrected by btrfs and both filesystems remain intact.
If you put btrfs on mdadm on AB drives, you combine the risk from both
vendors, lose both filesystems, and skew the failure statistics.

> We need to be better about this scenario, both in making sure we don't
> have bugs that contribute to the problem, but also that we have the
> tools necessary to recover when things go wrong.  Thanks,
> 
> Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-07  5:20                                                                                                                                                                                   ` Marc MERLIN
@ 2022-04-07 16:29                                                                                                                                                                                     ` Marc MERLIN
  2022-04-07 17:07                                                                                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-07 16:29 UTC (permalink / raw)
  To: Josef Bacik, Martin Steigerwald; +Cc: linux-btrfs

On Thu, Apr 07, 2022 at 09:30:41AM +0200, Martin Steigerwald wrote:
> I did not keep statistics, but this might be one of the longest threads 
> on BTRFS mailing list.
> 
> Good luck with restoring your filesystem or at least recovering all data 
> from it!

Haha, thanks. All props go to Josef for not giving up :)
Even if I don't get my data back without going to the backup, this will
have been a useful exercise, however I managed to hose my FS, made for
an interesting recovery case.

On Wed, Apr 06, 2022 at 10:20:22PM -0700, Marc MERLIN wrote:
> On Wed, Apr 06, 2022 at 09:37:17PM -0700, Marc MERLIN wrote:
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo
> 
> there may have been something else that ate all my memory.
> I killed shinobi and re-ran your code overnight while dumping
> memory. I'll have a memory trace running this time while it re-runs
> overnight.

Ok, good news is that it was actually shinobi that somehow managed to
take enough RAM to OOM hang the system, but in such a way that I had
never seen before (sysrq not even able to work anymore, which is
worrisome).  
btrfs-find-root just took another 8G which was enough to tip the glass
over, but shinobi (and ultimately the kernel for being unable to kill
userspace that takes too much RAM), was at fault.

Overnight, it ran stable at 8GB:
  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
31873 root      20   0 8024m 7.8g    0 R  99.9 25.2 667:57.24 btrfs-find-root

But the bad news is that it didn't complete, and it's not looking like it's 
converging.


gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1 2>&1 |tee /tmp/outo
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x56055fb76600
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x56055fb76600
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 62
fixed slot 88
fixed slot 235
fixed slot 237
Couldn't find a replacement block for slot 238
Couldn't find a replacement block for slot 243
fixed slot 33
fixed slot 35
fixed slot 32
fixed slot 133
fixed slot 136
fixed slot 207
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 484
Couldn't find a replacement block for slot 76
fixed slot 229
fixed slot 172
fixed slot 174
fixed slot 36
Couldn't find a replacement block for slot 20
fixed slot 24
Couldn't find a replacement block for slot 28
fixed slot 115
fixed slot 148
fixed slot 9
Couldn't find a replacement block for slot 318
fixed slot 245
fixed slot 1
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 134
fixed slot 159
fixed slot 170
fixed slot 204
fixed slot 149
fixed slot 185
fixed slot 257
fixed slot 46
fixed slot 60
fixed slot 91
fixed slot 92
fixed slot 95
fixed slot 247
fixed slot 105
Couldn't find a replacement block for slot 109
fixed slot 124
fixed slot 191
fixed slot 244
fixed slot 9
fixed slot 57
Couldn't find a replacement block for slot 166
fixed slot 163
fixed slot 78
fixed slot 179
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 218
fixed slot 26
fixed slot 75
Couldn't find a replacement block for slot 68
fixed slot 56
fixed slot 60
Couldn't find a replacement block for slot 231
fixed slot 141
fixed slot 151
fixed slot 156
fixed slot 159
fixed slot 163
fixed slot 171
fixed slot 200
fixed slot 5
Couldn't find a replacement block for slot 157
fixed slot 224
fixed slot 44
fixed slot 217
fixed slot 122
fixed slot 59
Couldn't find a replacement block for slot 120
fixed slot 181
fixed slot 187
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 220
fixed slot 60
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 192
fixed slot 9
fixed slot 7
Couldn't find a replacement block for slot 226
fixed slot 24
fixed slot 49
fixed slot 22
Couldn't find a replacement block for slot 186
fixed slot 149
fixed slot 150
Couldn't find a replacement block for slot 42
fixed slot 218
fixed slot 223
fixed slot 228
fixed slot 230
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 155
fixed slot 24
fixed slot 30
fixed slot 376
fixed slot 94
fixed slot 104
fixed slot 197
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 191
fixed slot 139
fixed slot 128
Couldn't find a replacement block for slot 143
fixed slot 16
fixed slot 17
fixed slot 114
fixed slot 28
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 83
fixed slot 86
fixed slot 92
fixed slot 149
fixed slot 117
Couldn't find a replacement block for slot 31
fixed slot 188
Couldn't find a replacement block for slot 102
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 79
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 21
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 276
Couldn't find a replacement block for slot 282
Couldn't find a replacement block for slot 283
Couldn't find a replacement block for slot 284
Couldn't find a replacement block for slot 285
Couldn't find a replacement block for slot 286
Couldn't find a replacement block for slot 287
Couldn't find a replacement block for slot 289
Couldn't find a replacement block for slot 290
Couldn't find a replacement block for slot 291
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 231
fixed slot 208
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 208
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 130
fixed slot 138
fixed slot 33
Couldn't find a replacement block for slot 179
fixed slot 154
Couldn't find a replacement block for slot 256
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 129
fixed slot 111
Couldn't find a replacement block for slot 103
Couldn't find a replacement block for slot 192
fixed slot 12
Couldn't find a replacement block for slot 205
fixed slot 24
fixed slot 28
fixed slot 31
fixed slot 38
fixed slot 47
fixed slot 62
fixed slot 75
fixed slot 86
fixed slot 89
fixed slot 93
fixed slot 108
Couldn't find a replacement block for slot 115
fixed slot 146
fixed slot 218
fixed slot 288
fixed slot 33
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 176
fixed slot 240
Couldn't find a replacement block for slot 241
fixed slot 34
fixed slot 35
Couldn't find a replacement block for slot 71
Couldn't find a replacement block for slot 140
fixed slot 213
Couldn't find a replacement block for slot 155
fixed slot 210
fixed slot 111
fixed slot 147
fixed slot 150
Couldn't find a replacement block for slot 234
Couldn't find a replacement block for slot 236
fixed slot 240
fixed slot 250
Couldn't find a replacement block for slot 259
fixed slot 269
fixed slot 283
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 48
fixed slot 135
fixed slot 192
fixed slot 206
fixed slot 43
fixed slot 111
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 45
fixed slot 47
fixed slot 212
fixed slot 49
fixed slot 190
fixed slot 4
fixed slot 40
fixed slot 110
fixed slot 227
fixed slot 290
fixed slot 296
fixed slot 297
fixed slot 302
fixed slot 349
fixed slot 357
Couldn't find a replacement block for slot 53
fixed slot 304
Couldn't find a replacement block for slot 328
fixed slot 376
fixed slot 54
fixed slot 125
fixed slot 195
Couldn't find a replacement block for slot 89
fixed slot 177
fixed slot 197
Couldn't find a replacement block for slot 219
fixed slot 233
fixed slot 36
fixed slot 52
fixed slot 61
fixed slot 71
fixed slot 113
fixed slot 119
fixed slot 138
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 200
fixed slot 215
Couldn't find a replacement block for slot 173
fixed slot 315
fixed slot 320
fixed slot 326
fixed slot 328
fixed slot 335
fixed slot 65
fixed slot 32
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 34
Couldn't find a replacement block for slot 37
fixed slot 40
Couldn't find a replacement block for slot 179
fixed slot 75
fixed slot 198
fixed slot 79
fixed slot 16
fixed slot 89
fixed slot 111
fixed slot 152
fixed slot 219
fixed slot 220
fixed slot 43
fixed slot 70
fixed slot 85
fixed slot 102
fixed slot 130
fixed slot 145
fixed slot 146
fixed slot 161
fixed slot 188
fixed slot 202
Couldn't find a replacement block for slot 9
fixed slot 62
fixed slot 122
fixed slot 128
fixed slot 137
fixed slot 142
fixed slot 216
Couldn't find a replacement block for slot 26
fixed slot 143
fixed slot 152
fixed slot 174
fixed slot 185
fixed slot 76
fixed slot 87
fixed slot 26
fixed slot 28
fixed slot 31
fixed slot 33
fixed slot 35
fixed slot 43
fixed slot 46
fixed slot 47
fixed slot 56
fixed slot 62
fixed slot 69
fixed slot 73
fixed slot 76
fixed slot 78
fixed slot 81
fixed slot 83
fixed slot 90
fixed slot 92
fixed slot 93
fixed slot 94
fixed slot 98
fixed slot 110
fixed slot 113
fixed slot 114
fixed slot 115
fixed slot 117
fixed slot 118
fixed slot 119
fixed slot 120
fixed slot 121
fixed slot 122
fixed slot 125
fixed slot 142
fixed slot 146
fixed slot 147
fixed slot 148
fixed slot 179
fixed slot 182
fixed slot 185
fixed slot 101
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 109
Couldn't find a replacement block for slot 430
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 41
fixed slot 63
fixed slot 33
fixed slot 36
Couldn't find a replacement block for slot 47
fixed slot 52
fixed slot 58
Couldn't find a replacement block for slot 426
Couldn't find a replacement block for slot 64
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 223
fixed slot 76
Couldn't find a replacement block for slot 53
Couldn't find a replacement block for slot 92
fixed slot 228
Couldn't find a replacement block for slot 109
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 173
Couldn't find a replacement block for slot 245
fixed slot 153
Couldn't find a replacement block for slot 115
Couldn't find a replacement block for slot 193
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 241
fixed slot 125
Couldn't find a replacement block for slot 98
Couldn't find a replacement block for slot 185
fixed slot 200
fixed slot 202
Couldn't find a replacement block for slot 203
fixed slot 204
Couldn't find a replacement block for slot 103
fixed slot 249
Couldn't find a replacement block for slot 250
fixed slot 284
fixed slot 305
fixed slot 128
fixed slot 144
Couldn't find a replacement block for slot 342
fixed slot 164
fixed slot 204
fixed slot 58
fixed slot 132
Couldn't find a replacement block for slot 161
fixed slot 177
fixed slot 209
Couldn't find a replacement block for slot 117
Couldn't find a replacement block for slot 248
Couldn't find a replacement block for slot 2
Couldn't find a replacement block for slot 96
fixed slot 5
fixed slot 114
Couldn't find a replacement block for slot 90
fixed slot 237
fixed slot 19
fixed slot 7
Couldn't find a replacement block for slot 27
fixed slot 48
fixed slot 53
Couldn't find a replacement block for slot 45
Couldn't find a replacement block for slot 225
fixed slot 50
Couldn't find a replacement block for slot 52
fixed slot 57
fixed slot 208
fixed slot 210
fixed slot 78
fixed slot 230
fixed slot 61
Couldn't find a replacement block for slot 81
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 174
fixed slot 96
fixed slot 63
fixed slot 113
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 26
Couldn't find a replacement block for slot 53
fixed slot 78
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 174
fixed slot 182
Couldn't find a replacement block for slot 183
fixed slot 184
Couldn't find a replacement block for slot 185
fixed slot 199
Couldn't find a replacement block for slot 201
Couldn't find a replacement block for slot 213
fixed slot 215
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 113
Couldn't find a replacement block for slot 120
Couldn't find a replacement block for slot 124
fixed slot 125
Couldn't find a replacement block for slot 127
Couldn't find a replacement block for slot 139
fixed slot 140
Couldn't find a replacement block for slot 148
Couldn't find a replacement block for slot 150
Couldn't find a replacement block for slot 156
Couldn't find a replacement block for slot 166
fixed slot 179
fixed slot 186
fixed slot 111
fixed slot 0
Couldn't find a replacement block for slot 9
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 44
fixed slot 49
Couldn't find a replacement block for slot 67
fixed slot 68
fixed slot 69
fixed slot 70
fixed slot 76
fixed slot 80
fixed slot 83
Couldn't find a replacement block for slot 86
fixed slot 113
fixed slot 140
fixed slot 143
fixed slot 144
fixed slot 184
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 194
Couldn't find a replacement block for slot 195
Couldn't find a replacement block for slot 196
fixed slot 197
Couldn't find a replacement block for slot 198
Couldn't find a replacement block for slot 199
fixed slot 202
Couldn't find a replacement block for slot 203
fixed slot 205
Couldn't find a replacement block for slot 215
Couldn't find a replacement block for slot 239
fixed slot 245
fixed slot 247
fixed slot 248
fixed slot 260
fixed slot 20
fixed slot 28
fixed slot 44
fixed slot 45
Couldn't find a replacement block for slot 48
Couldn't find a replacement block for slot 57
Couldn't find a replacement block for slot 61
Couldn't find a replacement block for slot 69
fixed slot 82
Couldn't find a replacement block for slot 86
fixed slot 90
Couldn't find a replacement block for slot 92
fixed slot 96
Couldn't find a replacement block for slot 97
fixed slot 110
fixed slot 116
fixed slot 121
fixed slot 138
fixed slot 150
fixed slot 158
fixed slot 160
fixed slot 162
Couldn't find a replacement block for slot 168
Couldn't find a replacement block for slot 169
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 173
fixed slot 175
fixed slot 179
fixed slot 181
Couldn't find a replacement block for slot 182
fixed slot 185
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 188
fixed slot 193
fixed slot 205
fixed slot 215
Couldn't find a replacement block for slot 224
fixed slot 233
fixed slot 237
fixed slot 243
fixed slot 256
Couldn't find a replacement block for slot 11
Couldn't find a replacement block for slot 15
fixed slot 21
fixed slot 26
Couldn't find a replacement block for slot 32
fixed slot 50
fixed slot 55
Couldn't find a replacement block for slot 57
fixed slot 112
fixed slot 113
fixed slot 114
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 127
fixed slot 131
Couldn't find a replacement block for slot 140
fixed slot 146
Couldn't find a replacement block for slot 153
fixed slot 154
Couldn't find a replacement block for slot 159
fixed slot 163
fixed slot 164
fixed slot 177
fixed slot 202
fixed slot 214
Couldn't find a replacement block for slot 220
fixed slot 230
Couldn't find a replacement block for slot 246
fixed slot 251
Couldn't find a replacement block for slot 259
Couldn't find a replacement block for slot 261
Couldn't find a replacement block for slot 262
Couldn't find a replacement block for slot 263
fixed slot 264
fixed slot 266
fixed slot 114
fixed slot 5
fixed slot 24
fixed slot 72
fixed slot 85
fixed slot 93
fixed slot 98
Couldn't find a replacement block for slot 111
fixed slot 128
fixed slot 130
Couldn't find a replacement block for slot 131
fixed slot 133
fixed slot 145
fixed slot 146
Couldn't find a replacement block for slot 75
Couldn't find a replacement block for slot 76
Couldn't find a replacement block for slot 77
Couldn't find a replacement block for slot 83
Couldn't find a replacement block for slot 141
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 37
Couldn't find a replacement block for slot 38
fixed slot 111
fixed slot 118
fixed slot 119
fixed slot 206
Couldn't find a replacement block for slot 226
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 229
Couldn't find a replacement block for slot 231
Couldn't find a replacement block for slot 374
Couldn't find a replacement block for slot 376
fixed slot 377
Couldn't find a replacement block for slot 60
fixed slot 122
Couldn't find a replacement block for slot 92
fixed slot 96
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 180
fixed slot 126
Couldn't find a replacement block for slot 119
Couldn't find a replacement block for slot 120
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 128
Couldn't find a replacement block for slot 129
Couldn't find a replacement block for slot 130
Couldn't find a replacement block for slot 131
Couldn't find a replacement block for slot 132
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 140
Couldn't find a replacement block for slot 141
fixed slot 129
fixed slot 134
Couldn't find a replacement block for slot 100
fixed slot 101
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 139
Couldn't find a replacement block for slot 35
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 367
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 244
Couldn't find a replacement block for slot 308
Couldn't find a replacement block for slot 309
Couldn't find a replacement block for slot 310
Couldn't find a replacement block for slot 311
Couldn't find a replacement block for slot 312
Couldn't find a replacement block for slot 326
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 49
fixed slot 13
Couldn't find a replacement block for slot 80
Couldn't find a replacement block for slot 145
Couldn't find a replacement block for slot 76
fixed slot 143
Couldn't find a replacement block for slot 28
fixed slot 159
Couldn't find a replacement block for slot 114
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 126
fixed slot 148
Couldn't find a replacement block for slot 150
fixed slot 155
fixed slot 161
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 204
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 211
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 220
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 199
fixed slot 161
Couldn't find a replacement block for slot 108
Couldn't find a replacement block for slot 110
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 159
Couldn't find a replacement block for slot 175
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 205
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 263
fixed slot 169
Couldn't find a replacement block for slot 18
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 49
Couldn't find a replacement block for slot 51
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 138
fixed slot 157
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 207
fixed slot 210
fixed slot 212
fixed slot 213
fixed slot 216
fixed slot 219
Couldn't find a replacement block for slot 235
fixed slot 242
fixed slot 243
fixed slot 244
Couldn't find a replacement block for slot 245
Couldn't find a replacement block for slot 269
fixed slot 170
Couldn't find a replacement block for slot 28
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 250
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 257
Couldn't find a replacement block for slot 265
Couldn't find a replacement block for slot 279
Couldn't find a replacement block for slot 280
Couldn't find a replacement block for slot 281
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 9
fixed slot 37
Couldn't find a replacement block for slot 40
Couldn't find a replacement block for slot 68
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 82
Couldn't find a replacement block for slot 90
Couldn't find a replacement block for slot 113
Couldn't find a replacement block for slot 114
fixed slot 115
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 117
Couldn't find a replacement block for slot 118
fixed slot 127
Couldn't find a replacement block for slot 135
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 179
Couldn't find a replacement block for slot 181
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 219
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 222
Couldn't find a replacement block for slot 235
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 242
Couldn't find a replacement block for slot 264
Couldn't find a replacement block for slot 265
Couldn't find a replacement block for slot 267
fixed slot 16
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 26
Couldn't find a replacement block for slot 41
fixed slot 44
Couldn't find a replacement block for slot 52
fixed slot 55
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 97
Couldn't find a replacement block for slot 98
fixed slot 100
Couldn't find a replacement block for slot 118
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 125
fixed slot 150
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 191
Couldn't find a replacement block for slot 198
Couldn't find a replacement block for slot 207
Couldn't find a replacement block for slot 208
fixed slot 211
fixed slot 213
Couldn't find a replacement block for slot 218
fixed slot 221
Couldn't find a replacement block for slot 231
fixed slot 248
Couldn't find a replacement block for slot 255
Couldn't find a replacement block for slot 256
fixed slot 173
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 5
fixed slot 28
Couldn't find a replacement block for slot 29
Couldn't find a replacement block for slot 30
fixed slot 31
fixed slot 39
Couldn't find a replacement block for slot 41
fixed slot 55
Couldn't find a replacement block for slot 78
Couldn't find a replacement block for slot 84
Couldn't find a replacement block for slot 98
Couldn't find a replacement block for slot 99
fixed slot 102
Couldn't find a replacement block for slot 114
Couldn't find a replacement block for slot 126
Couldn't find a replacement block for slot 128
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 139
Couldn't find a replacement block for slot 140
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 209
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 2(... still running)


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-07 16:29                                                                                                                                                                                     ` Marc MERLIN
@ 2022-04-07 17:07                                                                                                                                                                                       ` Josef Bacik
  2022-04-07 19:11                                                                                                                                                                                         ` Martin Steigerwald
  2022-04-07 22:09                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 2 replies; 479+ messages in thread
From: Josef Bacik @ 2022-04-07 17:07 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Martin Steigerwald, linux-btrfs

On Thu, Apr 7, 2022 at 12:29 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Apr 07, 2022 at 09:30:41AM +0200, Martin Steigerwald wrote:
> > I did not keep statistics, but this might be one of the longest threads
> > on BTRFS mailing list.
> >
> > Good luck with restoring your filesystem or at least recovering all data
> > from it!
>
> Haha, thanks. All props go to Josef for not giving up :)
> Even if I don't get my data back without going to the backup, this will
> have been a useful exercise, however I managed to hose my FS, made for
> an interesting recovery case.
>
> On Wed, Apr 06, 2022 at 10:20:22PM -0700, Marc MERLIN wrote:
> > On Wed, Apr 06, 2022 at 09:37:17PM -0700, Marc MERLIN wrote:
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo
> >
> > there may have been something else that ate all my memory.
> > I killed shinobi and re-ran your code overnight while dumping
> > memory. I'll have a memory trace running this time while it re-runs
> > overnight.
>
> Ok, good news is that it was actually shinobi that somehow managed to
> take enough RAM to OOM hang the system, but in such a way that I had
> never seen before (sysrq not even able to work anymore, which is
> worrisome).
> btrfs-find-root just took another 8G which was enough to tip the glass
> over, but shinobi (and ultimately the kernel for being unable to kill
> userspace that takes too much RAM), was at fault.
>
> Overnight, it ran stable at 8GB:
>   PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
> 31873 root      20   0 8024m 7.8g    0 R  99.9 25.2 667:57.24 btrfs-find-root
>
> But the bad news is that it didn't complete, and it's not looking like it's
> converging.
>

Ok let it keep going, I have an idea of how to make this less awful
but also not use all the memory.  I'm going to start writing this up
now, if I finish before your run finishes you can switch to the new
shit and see how that goes.  I've got meetings until 4pm my time (it's
1pm my time), so you're not going to hear from me for a while.
Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-07 17:07                                                                                                                                                                                       ` Josef Bacik
@ 2022-04-07 19:11                                                                                                                                                                                         ` Martin Steigerwald
  2022-04-07 22:09                                                                                                                                                                                         ` Josef Bacik
  1 sibling, 0 replies; 479+ messages in thread
From: Martin Steigerwald @ 2022-04-07 19:11 UTC (permalink / raw)
  To: Marc MERLIN, Josef Bacik; +Cc: linux-btrfs

Josef Bacik - 07.04.22, 19:07:14 CEST:
> Ok let it keep going, I have an idea of how to make this less awful
> but also not use all the memory.  I'm going to start writing this up
> now, if I finish before your run finishes you can switch to the new
> shit and see how that goes.  I've got meetings until 4pm my time (it's
> 1pm my time), so you're not going to hear from me for a while.

You guys are having fun with that, it seems.

-- 
Martin



^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-07 17:07                                                                                                                                                                                       ` Josef Bacik
  2022-04-07 19:11                                                                                                                                                                                         ` Martin Steigerwald
@ 2022-04-07 22:09                                                                                                                                                                                         ` Josef Bacik
  2022-04-07 22:33                                                                                                                                                                                           ` Marc MERLIN
  2022-04-08 10:22                                                                                                                                                                                           ` Marc MERLIN
  1 sibling, 2 replies; 479+ messages in thread
From: Josef Bacik @ 2022-04-07 22:09 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Martin Steigerwald, linux-btrfs

On Thu, Apr 7, 2022 at 1:07 PM Josef Bacik <josef@toxicpanda.com> wrote:
>
> On Thu, Apr 7, 2022 at 12:29 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Thu, Apr 07, 2022 at 09:30:41AM +0200, Martin Steigerwald wrote:
> > > I did not keep statistics, but this might be one of the longest threads
> > > on BTRFS mailing list.
> > >
> > > Good luck with restoring your filesystem or at least recovering all data
> > > from it!
> >
> > Haha, thanks. All props go to Josef for not giving up :)
> > Even if I don't get my data back without going to the backup, this will
> > have been a useful exercise, however I managed to hose my FS, made for
> > an interesting recovery case.
> >
> > On Wed, Apr 06, 2022 at 10:20:22PM -0700, Marc MERLIN wrote:
> > > On Wed, Apr 06, 2022 at 09:37:17PM -0700, Marc MERLIN wrote:
> > > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo
> > >
> > > there may have been something else that ate all my memory.
> > > I killed shinobi and re-ran your code overnight while dumping
> > > memory. I'll have a memory trace running this time while it re-runs
> > > overnight.
> >
> > Ok, good news is that it was actually shinobi that somehow managed to
> > take enough RAM to OOM hang the system, but in such a way that I had
> > never seen before (sysrq not even able to work anymore, which is
> > worrisome).
> > btrfs-find-root just took another 8G which was enough to tip the glass
> > over, but shinobi (and ultimately the kernel for being unable to kill
> > userspace that takes too much RAM), was at fault.
> >
> > Overnight, it ran stable at 8GB:
> >   PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
> > 31873 root      20   0 8024m 7.8g    0 R  99.9 25.2 667:57.24 btrfs-find-root
> >
> > But the bad news is that it didn't complete, and it's not looking like it's
> > converging.
> >
>
> Ok let it keep going, I have an idea of how to make this less awful
> but also not use all the memory.  I'm going to start writing this up
> now, if I finish before your run finishes you can switch to the new
> shit and see how that goes.  I've got meetings until 4pm my time (it's
> 1pm my time), so you're not going to hear from me for a while.
> Thanks,
>

Just following up on this, I've got hungry kids, I'm about halfway
through the new shit.  Depending on how much help kids need with
homework I may have this done later tonight, or it'll be tomorrow
morning.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-07 22:09                                                                                                                                                                                         ` Josef Bacik
@ 2022-04-07 22:33                                                                                                                                                                                           ` Marc MERLIN
  2022-04-08 10:22                                                                                                                                                                                           ` Marc MERLIN
  1 sibling, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-04-07 22:33 UTC (permalink / raw)
  To: Josef Bacik; +Cc: Martin Steigerwald, linux-btrfs

On Thu, Apr 07, 2022 at 06:09:53PM -0400, Josef Bacik wrote:
> Just following up on this, I've got hungry kids, I'm about halfway
> through the new shit.  Depending on how much help kids need with
> homework I may have this done later tonight, or it'll be tomorrow
> morning.  Thanks,

it's stlil churning along and looping. I have no idea if it's
converging, or not.


parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x56055fb76600
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x56055fb76600
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 62
fixed slot 88
fixed slot 235
fixed slot 237
Couldn't find a replacement block for slot 238
Couldn't find a replacement block for slot 243
fixed slot 33
fixed slot 35
fixed slot 32
fixed slot 133
fixed slot 136
fixed slot 207
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 484
Couldn't find a replacement block for slot 76
fixed slot 229
fixed slot 172
fixed slot 174
fixed slot 36
Couldn't find a replacement block for slot 20
fixed slot 24
Couldn't find a replacement block for slot 28
fixed slot 115
fixed slot 148
fixed slot 9
Couldn't find a replacement block for slot 318
fixed slot 245
fixed slot 1
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 134
fixed slot 159
fixed slot 170
fixed slot 204
fixed slot 149
fixed slot 185
fixed slot 257
fixed slot 46
fixed slot 60
fixed slot 91
fixed slot 92
fixed slot 95
fixed slot 247
fixed slot 105
Couldn't find a replacement block for slot 109
fixed slot 124
fixed slot 191
fixed slot 244
fixed slot 9
fixed slot 57
Couldn't find a replacement block for slot 166
fixed slot 163
fixed slot 78
fixed slot 179
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 218
fixed slot 26
fixed slot 75
Couldn't find a replacement block for slot 68
fixed slot 56
fixed slot 60
Couldn't find a replacement block for slot 231
fixed slot 141
fixed slot 151
fixed slot 156
fixed slot 159
fixed slot 163
fixed slot 171
fixed slot 200
fixed slot 5
Couldn't find a replacement block for slot 157
fixed slot 224
fixed slot 44
fixed slot 217
fixed slot 122
fixed slot 59
Couldn't find a replacement block for slot 120
fixed slot 181
fixed slot 187
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 220
fixed slot 60
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 192
fixed slot 9
fixed slot 7
Couldn't find a replacement block for slot 226
fixed slot 24
fixed slot 49
fixed slot 22
Couldn't find a replacement block for slot 186
fixed slot 149
fixed slot 150
Couldn't find a replacement block for slot 42
fixed slot 218
fixed slot 223
fixed slot 228
fixed slot 230
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 155
fixed slot 24
fixed slot 30
fixed slot 376
fixed slot 94
fixed slot 104
fixed slot 197
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 191
fixed slot 139
fixed slot 128
Couldn't find a replacement block for slot 143
fixed slot 16
fixed slot 17
fixed slot 114
fixed slot 28
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 83
fixed slot 86
fixed slot 92
fixed slot 149
fixed slot 117
Couldn't find a replacement block for slot 31
fixed slot 188
Couldn't find a replacement block for slot 102
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 79
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 21
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 276
Couldn't find a replacement block for slot 282
Couldn't find a replacement block for slot 283
Couldn't find a replacement block for slot 284
Couldn't find a replacement block for slot 285
Couldn't find a replacement block for slot 286
Couldn't find a replacement block for slot 287
Couldn't find a replacement block for slot 289
Couldn't find a replacement block for slot 290
Couldn't find a replacement block for slot 291
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 231
fixed slot 208
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 208
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 130
fixed slot 138
fixed slot 33
Couldn't find a replacement block for slot 179
fixed slot 154
Couldn't find a replacement block for slot 256
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 129
fixed slot 111
Couldn't find a replacement block for slot 103
Couldn't find a replacement block for slot 192
fixed slot 12
Couldn't find a replacement block for slot 205
fixed slot 24
fixed slot 28
fixed slot 31
fixed slot 38
fixed slot 47
fixed slot 62
fixed slot 75
fixed slot 86
fixed slot 89
fixed slot 93
fixed slot 108
Couldn't find a replacement block for slot 115
fixed slot 146
fixed slot 218
fixed slot 288
fixed slot 33
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 176
fixed slot 240
Couldn't find a replacement block for slot 241
fixed slot 34
fixed slot 35
Couldn't find a replacement block for slot 71
Couldn't find a replacement block for slot 140
fixed slot 213
Couldn't find a replacement block for slot 155
fixed slot 210
fixed slot 111
fixed slot 147
fixed slot 150
Couldn't find a replacement block for slot 234
Couldn't find a replacement block for slot 236
fixed slot 240
fixed slot 250
Couldn't find a replacement block for slot 259
fixed slot 269
fixed slot 283
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 48
fixed slot 135
fixed slot 192
fixed slot 206
fixed slot 43
fixed slot 111
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 45
fixed slot 47
fixed slot 212
fixed slot 49
fixed slot 190
fixed slot 4
fixed slot 40
fixed slot 110
fixed slot 227
fixed slot 290
fixed slot 296
fixed slot 297
fixed slot 302
fixed slot 349
fixed slot 357
Couldn't find a replacement block for slot 53
fixed slot 304
Couldn't find a replacement block for slot 328
fixed slot 376
fixed slot 54
fixed slot 125
fixed slot 195
Couldn't find a replacement block for slot 89
fixed slot 177
fixed slot 197
Couldn't find a replacement block for slot 219
fixed slot 233
fixed slot 36
fixed slot 52
fixed slot 61
fixed slot 71
fixed slot 113
fixed slot 119
fixed slot 138
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 200
fixed slot 215
Couldn't find a replacement block for slot 173
fixed slot 315
fixed slot 320
fixed slot 326
fixed slot 328
fixed slot 335
fixed slot 65
fixed slot 32
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 34
Couldn't find a replacement block for slot 37
fixed slot 40
Couldn't find a replacement block for slot 179
fixed slot 75
fixed slot 198
fixed slot 79
fixed slot 16
fixed slot 89
fixed slot 111
fixed slot 152
fixed slot 219
fixed slot 220
fixed slot 43
fixed slot 70
fixed slot 85
fixed slot 102
fixed slot 130
fixed slot 145
fixed slot 146
fixed slot 161
fixed slot 188
fixed slot 202
Couldn't find a replacement block for slot 9
fixed slot 62
fixed slot 122
fixed slot 128
fixed slot 137
fixed slot 142
fixed slot 216
Couldn't find a replacement block for slot 26
fixed slot 143
fixed slot 152
fixed slot 174
fixed slot 185
fixed slot 76
fixed slot 87
fixed slot 26
fixed slot 28
fixed slot 31
fixed slot 33
fixed slot 35
fixed slot 43
fixed slot 46
fixed slot 47
fixed slot 56
fixed slot 62
fixed slot 69
fixed slot 73
fixed slot 76
fixed slot 78
fixed slot 81
fixed slot 83
fixed slot 90
fixed slot 92
fixed slot 93
fixed slot 94
fixed slot 98
fixed slot 110
fixed slot 113
fixed slot 114
fixed slot 115
fixed slot 117
fixed slot 118
fixed slot 119
fixed slot 120
fixed slot 121
fixed slot 122
fixed slot 125
fixed slot 142
fixed slot 146
fixed slot 147
fixed slot 148
fixed slot 179
fixed slot 182
fixed slot 185
fixed slot 101
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 109
Couldn't find a replacement block for slot 430
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 41
fixed slot 63
fixed slot 33
fixed slot 36
Couldn't find a replacement block for slot 47
fixed slot 52
fixed slot 58
Couldn't find a replacement block for slot 426
Couldn't find a replacement block for slot 64
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 223
fixed slot 76
Couldn't find a replacement block for slot 53
Couldn't find a replacement block for slot 92
fixed slot 228
Couldn't find a replacement block for slot 109
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 173
Couldn't find a replacement block for slot 245
fixed slot 153
Couldn't find a replacement block for slot 115
Couldn't find a replacement block for slot 193
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 241
fixed slot 125
Couldn't find a replacement block for slot 98
Couldn't find a replacement block for slot 185
fixed slot 200
fixed slot 202
Couldn't find a replacement block for slot 203
fixed slot 204
Couldn't find a replacement block for slot 103
fixed slot 249
Couldn't find a replacement block for slot 250
fixed slot 284
fixed slot 305
fixed slot 128
fixed slot 144
Couldn't find a replacement block for slot 342
fixed slot 164
fixed slot 204
fixed slot 58
fixed slot 132
Couldn't find a replacement block for slot 161
fixed slot 177
fixed slot 209
Couldn't find a replacement block for slot 117
Couldn't find a replacement block for slot 248
Couldn't find a replacement block for slot 2
Couldn't find a replacement block for slot 96
fixed slot 5
fixed slot 114
Couldn't find a replacement block for slot 90
fixed slot 237
fixed slot 19
fixed slot 7
Couldn't find a replacement block for slot 27
fixed slot 48
fixed slot 53
Couldn't find a replacement block for slot 45
Couldn't find a replacement block for slot 225
fixed slot 50
Couldn't find a replacement block for slot 52
fixed slot 57
fixed slot 208
fixed slot 210
fixed slot 78
fixed slot 230
fixed slot 61
Couldn't find a replacement block for slot 81
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 174
fixed slot 96
fixed slot 63
fixed slot 113
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 26
Couldn't find a replacement block for slot 53
fixed slot 78
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 174
fixed slot 182
Couldn't find a replacement block for slot 183
fixed slot 184
Couldn't find a replacement block for slot 185
fixed slot 199
Couldn't find a replacement block for slot 201
Couldn't find a replacement block for slot 213
fixed slot 215
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 113
Couldn't find a replacement block for slot 120
Couldn't find a replacement block for slot 124
fixed slot 125
Couldn't find a replacement block for slot 127
Couldn't find a replacement block for slot 139
fixed slot 140
Couldn't find a replacement block for slot 148
Couldn't find a replacement block for slot 150
Couldn't find a replacement block for slot 156
Couldn't find a replacement block for slot 166
fixed slot 179
fixed slot 186
fixed slot 111
fixed slot 0
Couldn't find a replacement block for slot 9
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 44
fixed slot 49
Couldn't find a replacement block for slot 67
fixed slot 68
fixed slot 69
fixed slot 70
fixed slot 76
fixed slot 80
fixed slot 83
Couldn't find a replacement block for slot 86
fixed slot 113
fixed slot 140
fixed slot 143
fixed slot 144
fixed slot 184
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 194
Couldn't find a replacement block for slot 195
Couldn't find a replacement block for slot 196
fixed slot 197
Couldn't find a replacement block for slot 198
Couldn't find a replacement block for slot 199
fixed slot 202
Couldn't find a replacement block for slot 203
fixed slot 205
Couldn't find a replacement block for slot 215
Couldn't find a replacement block for slot 239
fixed slot 245
fixed slot 247
fixed slot 248
fixed slot 260
fixed slot 20
fixed slot 28
fixed slot 44
fixed slot 45
Couldn't find a replacement block for slot 48
Couldn't find a replacement block for slot 57
Couldn't find a replacement block for slot 61
Couldn't find a replacement block for slot 69
fixed slot 82
Couldn't find a replacement block for slot 86
fixed slot 90
Couldn't find a replacement block for slot 92
fixed slot 96
Couldn't find a replacement block for slot 97
fixed slot 110
fixed slot 116
fixed slot 121
fixed slot 138
fixed slot 150
fixed slot 158
fixed slot 160
fixed slot 162
Couldn't find a replacement block for slot 168
Couldn't find a replacement block for slot 169
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 173
fixed slot 175
fixed slot 179
fixed slot 181
Couldn't find a replacement block for slot 182
fixed slot 185
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 188
fixed slot 193
fixed slot 205
fixed slot 215
Couldn't find a replacement block for slot 224
fixed slot 233
fixed slot 237
fixed slot 243
fixed slot 256
Couldn't find a replacement block for slot 11
Couldn't find a replacement block for slot 15
fixed slot 21
fixed slot 26
Couldn't find a replacement block for slot 32
fixed slot 50
fixed slot 55
Couldn't find a replacement block for slot 57
fixed slot 112
fixed slot 113
fixed slot 114
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 127
fixed slot 131
Couldn't find a replacement block for slot 140
fixed slot 146
Couldn't find a replacement block for slot 153
fixed slot 154
Couldn't find a replacement block for slot 159
fixed slot 163
fixed slot 164
fixed slot 177
fixed slot 202
fixed slot 214
Couldn't find a replacement block for slot 220
fixed slot 230
Couldn't find a replacement block for slot 246
fixed slot 251
Couldn't find a replacement block for slot 259
Couldn't find a replacement block for slot 261
Couldn't find a replacement block for slot 262
Couldn't find a replacement block for slot 263
fixed slot 264
fixed slot 266
fixed slot 114
fixed slot 5
fixed slot 24
fixed slot 72
fixed slot 85
fixed slot 93
fixed slot 98
Couldn't find a replacement block for slot 111
fixed slot 128
fixed slot 130
Couldn't find a replacement block for slot 131
fixed slot 133
fixed slot 145
fixed slot 146
Couldn't find a replacement block for slot 75
Couldn't find a replacement block for slot 76
Couldn't find a replacement block for slot 77
Couldn't find a replacement block for slot 83
Couldn't find a replacement block for slot 141
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 37
Couldn't find a replacement block for slot 38
fixed slot 111
fixed slot 118
fixed slot 119
fixed slot 206
Couldn't find a replacement block for slot 226
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 229
Couldn't find a replacement block for slot 231
Couldn't find a replacement block for slot 374
Couldn't find a replacement block for slot 376
fixed slot 377
Couldn't find a replacement block for slot 60
fixed slot 122
Couldn't find a replacement block for slot 92
fixed slot 96
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 180
fixed slot 126
Couldn't find a replacement block for slot 119
Couldn't find a replacement block for slot 120
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 128
Couldn't find a replacement block for slot 129
Couldn't find a replacement block for slot 130
Couldn't find a replacement block for slot 131
Couldn't find a replacement block for slot 132
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 140
Couldn't find a replacement block for slot 141
fixed slot 129
fixed slot 134
Couldn't find a replacement block for slot 100
fixed slot 101
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 139
Couldn't find a replacement block for slot 35
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 367
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 244
Couldn't find a replacement block for slot 308
Couldn't find a replacement block for slot 309
Couldn't find a replacement block for slot 310
Couldn't find a replacement block for slot 311
Couldn't find a replacement block for slot 312
Couldn't find a replacement block for slot 326
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 49
fixed slot 13
Couldn't find a replacement block for slot 80
Couldn't find a replacement block for slot 145
Couldn't find a replacement block for slot 76
fixed slot 143
Couldn't find a replacement block for slot 28
fixed slot 159
Couldn't find a replacement block for slot 114
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 126
fixed slot 148
Couldn't find a replacement block for slot 150
fixed slot 155
fixed slot 161
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 204
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 211
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 220
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 199
fixed slot 161
Couldn't find a replacement block for slot 108
Couldn't find a replacement block for slot 110
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 159
Couldn't find a replacement block for slot 175
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 205
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 263
fixed slot 169
Couldn't find a replacement block for slot 18
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 49
Couldn't find a replacement block for slot 51
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 138
fixed slot 157
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 207
fixed slot 210
fixed slot 212
fixed slot 213
fixed slot 216
fixed slot 219
Couldn't find a replacement block for slot 235
fixed slot 242
fixed slot 243
fixed slot 244
Couldn't find a replacement block for slot 245
Couldn't find a replacement block for slot 269
fixed slot 170
Couldn't find a replacement block for slot 28
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 250
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 257
Couldn't find a replacement block for slot 265
Couldn't find a replacement block for slot 279
Couldn't find a replacement block for slot 280
Couldn't find a replacement block for slot 281
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 9
fixed slot 37
Couldn't find a replacement block for slot 40
Couldn't find a replacement block for slot 68
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 82
Couldn't find a replacement block for slot 90
Couldn't find a replacement block for slot 113
Couldn't find a replacement block for slot 114
fixed slot 115
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 117
Couldn't find a replacement block for slot 118
fixed slot 127
Couldn't find a replacement block for slot 135
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 179
Couldn't find a replacement block for slot 181
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 219
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 222
Couldn't find a replacement block for slot 235
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 242
Couldn't find a replacement block for slot 264
Couldn't find a replacement block for slot 265
Couldn't find a replacement block for slot 267
fixed slot 16
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 26
Couldn't find a replacement block for slot 41
fixed slot 44
Couldn't find a replacement block for slot 52
fixed slot 55
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 97
Couldn't find a replacement block for slot 98
fixed slot 100
Couldn't find a replacement block for slot 118
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 125
fixed slot 150
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 191
Couldn't find a replacement block for slot 198
Couldn't find a replacement block for slot 207
Couldn't find a replacement block for slot 208
fixed slot 211
fixed slot 213
Couldn't find a replacement block for slot 218
fixed slot 221
Couldn't find a replacement block for slot 231
fixed slot 248
Couldn't find a replacement block for slot 255
Couldn't find a replacement block for slot 256
fixed slot 173
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 5
fixed slot 28
Couldn't find a replacement block for slot 29
Couldn't find a replacement block for slot 30
fixed slot 31
fixed slot 39
Couldn't find a replacement block for slot 41
fixed slot 55
Couldn't find a replacement block for slot 78
Couldn't find a replacement block for slot 84
Couldn't find a replacement block for slot 98
Couldn't find a replacement block for slot 99
fixed slot 102
Couldn't find a replacement block for slot 114
Couldn't find a replacement block for slot 126
Couldn't find a replacement block for slot 128
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 139
Couldn't find a replacement block for slot 140
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 209
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 211
Couldn't find a replacement block for slot 213
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 222
Couldn't find a replacement block for slot 223
Couldn't find a replacement block for slot 224
Couldn't find a replacement block for slot 225
Couldn't find a replacement block for slot 226
Couldn't find a replacement block for slot 227
Couldn't find a replacement block for slot 229
Couldn't find a replacement block for slot 230
Couldn't find a replacement block for slot 231
Couldn't find a replacement block for slot 233
Couldn't find a replacement block for slot 252
Couldn't find a replacement block for slot 253
Couldn't find a replacement block for slot 254
fixed slot 256
Couldn't find a replacement block for slot 11
fixed slot 28
Couldn't find a replacement block for slot 33
fixed slot 39
Couldn't find a replacement block for slot 63
Couldn't find a replacement block for slot 72
fixed slot 119
Couldn't find a replacement block for slot 146
fixed slot 156
Couldn't find a replacement block for slot 179
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 193
Couldn't find a replacement block for slot 194
Couldn't find a replacement block for slot 197
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 250
Couldn't find a replacement block for slot 252
Couldn't find a replacement block for slot 5
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 18
Couldn't find a replacement block for slot 31
Couldn't find a replacement block for slot 53
Couldn't find a replacement block for slot 56
Couldn't find a replacement block for slot 57
Couldn't find a replacement block for slot 62
Couldn't find a replacement block for slot 84
Couldn't find a replacement block for slot 85
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 87
fixed slot 99
Couldn't find a replacement block for slot 100
Couldn't find a replacement block for slot 101
fixed slot 108
fixed slot 141
fixed slot 155
fixed slot 178
Couldn't find a replacement block for slot 181
Couldn't find a replacement block for slot 197
fixed slot 177
fixed slot 193
Couldn't find a replacement block for slot 194
Couldn't find a replacement block for slot 195
fixed slot 226
Couldn't find a replacement block for slot 273
fixed slot 302
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 70
Couldn't find a replacement block for slot 143
Couldn't find a replacement block for slot 144
fixed slot 204
Couldn't find a replacement block for slot 129
Couldn't find a replacement block for slot 130
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 255
Couldn't find a replacement block for slot 214
fixed slot 248
fixed slot 220
fixed slot 204
Couldn't find a replacement block for slot 205
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 207
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 211
Couldn't find a replacement block for slot 215
Couldn't find a replacement block for slot 220
fixed slot 234
Couldn't find a replacement block for slot 235
fixed slot 236
Couldn't find a replacement block for slot 247
Couldn't find a replacement block for slot 248
Couldn't find a replacement block for slot 267
fixed slot 276
fixed slot 288
Couldn't find a replacement block for slot 317
fixed slot 330
Couldn't find a replacement block for slot 337
Couldn't find a replacement block for slot 345
Couldn't find a replacement block for slot 346
fixed slot 361
Couldn't find a replacement block for slot 365
Couldn't find a replacement block for slot 366
Couldn't find a replacement block for slot 369
Couldn't find a replacement block for slot 370
Couldn't find a replacement block for slot 372
Couldn't find a replacement block for slot 373
fixed slot 379
Couldn't find a replacement block for slot 380
Couldn't find a replacement block for slot 381
Couldn't find a replacement block for slot 382
Couldn't find a replacement block for slot 387
Couldn't find a replacement block for slot 388
Couldn't find a replacement block for slot 389
Couldn't find a replacement block for slot 390
Couldn't find a replacement block for slot 394
fixed slot 395
Couldn't find a replacement block for slot 396
fixed slot 407
Couldn't find a replacement block for slot 408
Couldn't find a replacement block for slot 410
Couldn't find a replacement block for slot 411
Couldn't find a replacement block for slot 412
Couldn't find a replacement block for slot 425
Couldn't find a replacement block for slot 426
Couldn't find a replacement block for slot 427
Couldn't find a replacement block for slot 428
fixed slot 221
fixed slot 0
Couldn't find a replacement block for slot 1
Couldn't find a replacement block for slot 2
Couldn't find a replacement block for slot 8
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 16
Couldn't find a replacement block for slot 17
Couldn't find a replacement block for slot 20
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 38
Couldn't find a replacement block for slot 48
Couldn't find a replacement block for slot 49
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 54
Couldn't find a replacement block for slot 55
Couldn't find a replacement block for slot 56
fixed slot 67
fixed slot 83
Couldn't find a replacement block for slot 84
Couldn't find a replacement block for slot 119
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 126
fixed slot 206
Couldn't find a replacement block for slot 207
Couldn't find a replacement block for slot 220
fixed slot 224
fixed slot 143
fixed slot 158
Couldn't find a replacement block for slot 159
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 204
Couldn't find a replacement block for slot 227
Couldn't find a replacement block for slot 228
fixed slot 229
fixed slot 225
fixed slot 95
fixed slot 104
Couldn't find a replacement block for slot 105
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 131
Couldn't find a replacement block for slot 141
Couldn't find a replacement block for slot 163
Couldn't find a replacement block for slot 165
fixed slot 171
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 203
fixed slot 205
Couldn't find a replacement block for slot 213
Couldn't find a replacement block for slot 215
fixed slot 221
Couldn't find a replacement block for slot 225
fixed slot 226
fixed slot 231
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 242
fixed slot 243
fixed slot 254
Couldn't find a replacement block for slot 255
Couldn't find a replacement block for slot 257
Couldn't find a replacement block for slot 258
Couldn't find a replacement block for slot 229
fixed slot 233
Couldn't find a replacement block for slot 147
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 154
fixed slot 155
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 183
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 188
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 191
Couldn't find a replacement block for slot 192
Couldn't find a replacement block for slot 193
Couldn't find a replacement block for slot 201
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 233
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 242
Couldn't find a replacement block for slot 1
Couldn't find a replacement block for slot 2
Couldn't find a replacement block for slot 3
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 12
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 14
Couldn't find a replacement block for slot 19
Couldn't find a replacement block for slot 22
fixed slot 45
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 47
Couldn't find a replacement block for slot 48
Couldn't find a replacement block for slot 54
Couldn't find a replacement block for slot 59
Couldn't find a replacement block for slot 61
Couldn't find a replacement block for slot 76
Couldn't find a replacement block for slot 78
fixed slot 81
Couldn't find a replacement block for slot 85
Couldn't find a replacement block for slot 87
fixed slot 90
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 92
fixed slot 94
Couldn't find a replacement block for slot 104
fixed slot 170
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 208
fixed slot 210
fixed slot 218
Couldn't find a replacement block for slot 256
Couldn't find a replacement block for slot 271
fixed slot 235
fixed slot 14
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 16
Couldn't find a replacement block for slot 17
Couldn't find a replacement block for slot 24
Couldn't find a replacement block for slot 25
Couldn't find a replacement block for slot 233
fixed slot 245
Couldn't find a replacement block for slot 246
Couldn't find a replacement block for slot 247
Couldn't find a replacement block for slot 248
Couldn't find a replacement block for slot 249
Couldn't find a replacement block for slot 285
fixed slot 286
Couldn't find a replacement block for slot 287
fixed slot 49
Couldn't find a replacement block for slot 53
fixed slot 63
Couldn't find a replacement block for slot 67
Couldn't find a replacement block for slot 68
Couldn't find a replacement block for slot 70
fixed slot 78
Couldn't find a replacement block for slot 101
Couldn't find a replacement block for slot 102
fixed slot 105
Couldn't find a replacement block for slot 114
fixed slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 131
Couldn't find a replacement block for slot 133
Couldn't find a replacement block for slot 142
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 154
Couldn't find a replacement block for slot 155
fixed slot 179
fixed slot 196
fixed slot 199
Couldn't find a replacement block for slot 225
Couldn't find a replacement block for slot 259
Couldn't find a replacement block for slot 260
Couldn't find a replacement block for slot 261
fixed slot 29
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 209
Couldn't find a replacement block for slot 429
fixed slot 289
Couldn't find a replacement block for slot 195
fixed slot 172
Couldn't find a replacement block for slot 173
Couldn't find a replacement block for slot 295
fixed slot 4
fixed slot 7
Couldn't find a replacement block for slot 11
Couldn't find a replacement block for slot 14
Couldn't find a replacement block for slot 15
fixed slot 16
Couldn't find a replacement block for slot 35
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 38
Couldn't find a replacement block for slot 40
Couldn't find a replacement block for slot 41
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 47
Couldn't find a replacement block for slot 48
fixed slot 49
Couldn't find a replacement block for slot 154
Couldn't find a replacement block for slot 253
fixed slot 7
Couldn't find a replacement block for slot 189
Couldn't find a replacement block for slot 191
Couldn't find a replacement block for slot 62
Couldn't find a replacement block for slot 63
Couldn't find a replacement block for slot 168
fixed slot 172
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 214
fixed slot 267
fixed slot 268
Couldn't find a replacement block for slot 269
Couldn't find a replacement block for slot 270
Couldn't find a replacement block for slot 280
fixed slot 281
Couldn't find a replacement block for slot 282
fixed slot 13
Couldn't find a replacement block for slot 30
Couldn't find a replacement block for slot 31
Couldn't find a replacement block for slot 69
fixed slot 70
Couldn't find a replacement block for slot 90
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 130
Couldn't find a replacement block for slot 146
Couldn't find a replacement block for slot 147
Couldn't find a replacement block for slot 156
fixed slot 173
Couldn't find a replacement block for slot 174
Couldn't find a replacement block for slot 175
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 188
Couldn't find a replacement block for slot 209
Couldn't find a replacement block for slot 220
Couldn't find a replacement block for slot 221
fixed slot 229
Couldn't find a replacement block for slot 230
Couldn't find a replacement block for slot 231
Couldn't find a replacement block for slot 233
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 244
fixed slot 3
fixed slot 4
Couldn't find a replacement block for slot 5
Couldn't find a replacement block for slot 7
Couldn't find a replacement block for slot 8
Couldn't find a replacement block for slot 9
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 14
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 16
Couldn't find a replacement block for slot 17
Couldn't find a replacement block for slot 18
fixed slot 25
Couldn't find a replacement block for slot 28
Couldn't find a replacement block for slot 40
fixed slot 41
Couldn't find a replacement block for slot 42
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 44
Couldn't find a replacement block for slot 47
Couldn't find a replacement block for slot 55
fixed slot 60
Couldn't find a replacement block for slot 61
Couldn't find a replacement block for slot 63
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 73
Couldn't find a replacement block for slot 74
fixed slot 78
Couldn't find a replacement block for slot 80
Couldn't find a replacement block for slot 82
Couldn't find a replacement block for slot 92
Couldn't find a replacement block for slot 93
Couldn't find a replacement block for slot 99
fixed slot 103
Couldn't find a replacement block for slot 104
Couldn't find a replacement block for slot 106
fixed slot 109
Couldn't find a replacement block for slot 110
fixed slot 111
fixed slot 120
Couldn't find a replacement block for slot 121
fixed slot 125
Couldn't find a replacement block for slot 126
Couldn't find a replacement block for slot 127
Couldn't find a replacement block for slot 128
fixed slot 301
Couldn't find a replacement block for slot 19
fixed slot 22
fixed slot 28
Couldn't find a replacement block for slot 29
Couldn't find a replacement block for slot 30
fixed slot 31
Couldn't find a replacement block for slot 38
Couldn't find a replacement block for slot 39
fixed slot 48
Couldn't find a replacement block for slot 51
fixed slot 53
Couldn't find a replacement block for slot 55
Couldn't find a replacement block for slot 56
fixed slot 62
fixed slot 77
Couldn't find a replacement block for slot 78
Couldn't find a replacement bl (waiting for more output)
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-07 22:09                                                                                                                                                                                         ` Josef Bacik
  2022-04-07 22:33                                                                                                                                                                                           ` Marc MERLIN
@ 2022-04-08 10:22                                                                                                                                                                                           ` Marc MERLIN
  2022-04-08 10:23                                                                                                                                                                                             ` Josef Bacik
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-08 10:22 UTC (permalink / raw)
  To: Josef Bacik; +Cc: Martin Steigerwald, linux-btrfs

On Thu, Apr 07, 2022 at 06:09:53PM -0400, Josef Bacik wrote:
> Just following up on this, I've got hungry kids, I'm about halfway
> through the new shit.  Depending on how much help kids need with
> homework I may have this done later tonight, or it'll be tomorrow
> morning.  Thanks,

It finished after more than 24H, but that didn't seem to be enough, quite.
Hopefully we're getting closer, though :)

Couldn't find a replacement block for slot 361
Couldn't find a replacement block for slot 363
fixed slot 372
Couldn't find a replacement block for slot 374
Couldn't find a replacement block for slot 375
Couldn't find a replacement block for slot 403
Couldn't find a replacement block for slot 404
Couldn't find a replacement block for slot 405
Couldn't find a replacement block for slot 406
Couldn't find a replacement block for slot 407
Couldn't find a replacement block for slot 408
Couldn't find a replacement block for slot 409
Couldn't find a replacement block for slot 413
Couldn't find a replacement block for slot 414
Couldn't find a replacement block for slot 415
Couldn't find a replacement block for slot 416
Couldn't find a replacement block for slot 417
Couldn't find a replacement block for slot 418
Couldn't find a replacement block for slot 419
Couldn't find a replacement block for slot 420
Couldn't find a replacement block for slot 421
Couldn't find a replacement block for slot 422
Couldn't find a replacement block for slot 423
Couldn't find a replacement block for slot 424
Couldn't find a replacement block for slot 427
Couldn't find a replacement block for slot 437
Couldn't find a replacement block for slot 438
Couldn't find a replacement block for slot 443
Couldn't find a replacement block for slot 444
Couldn't find a replacement block for slot 445
fixed slot 446
fixed slot 447
gargamel:/var/local/src/btrfs-progs-josefbacik# 


gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree --repair /dev/mapper/dshelf1  
enabling repair mode
WARNING:

	Do not use --repair unless you are advised to do so by a developer
	or an experienced user, and then only after having accepted that no
	fsck can successfully repair all types of filesystem corruption. Eg.
	some software or hardware bugs can fatally damage a volume.
	The operation will start in 10 seconds.
	Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x5641ceb95fd0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x5641ceb95fd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Creating a new extent tree
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6
checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d
checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188
checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858
checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82
checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed
checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c
checksum verify failed on 15645178363904 wanted 0xba219c04 found 0x86a9d7f0
checksum verify failed on 15645917708288 wanted 0x4dab2011 found 0xb6299718
checksum verify failed on 15645365747712 wanted 0x3bdf1829 found 0xdbfcbd25
checksum verify failed on 15645365747712 wanted 0x4f4dcd5c found 0xa1d136c1
checksum verify failed on 15645365747712 wanted 0x4f4dcd5c found 0xa1d136c1
bad tree block 15645365747712, bad level, 240 > 8
Error reading tree block owner 7
error pinning down used bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13577814573056 len 16384
gargamel:/var/local/src/btrfs-progs-josefbacik# 


gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro /dev/mapper/dshelf1 /mnt/mnt
mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error.
gargamel:/var/local/src/btrfs-progs-josefbacik# dmtail
[106691.309851] BTRFS info (device dm-1): disk space caching is enabled
[106691.363587] BTRFS info (device dm-1): has skinny extents
[106691.394712] verify_parent_transid: 4 callbacks suppressed
[106691.394716] BTRFS error (device dm-1): parent transid verify failed on 22216704 wanted 1600938 found 1602177
[106691.443339] BTRFS error (device dm-1): parent transid verify failed on 22216704 wanted 1600938 found 1602177
[106691.474616] BTRFS error (device dm-1): failed to read chunk tree: -5
[106691.501647] BTRFS error (device dm-1): open_ctree failed
[106691.519613] systemd-udevd[31783]: btrfs-19: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-08 10:22                                                                                                                                                                                           ` Marc MERLIN
@ 2022-04-08 10:23                                                                                                                                                                                             ` Josef Bacik
  2022-04-08 20:09                                                                                                                                                                                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-08 10:23 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Martin Steigerwald, linux-btrfs

On Fri, Apr 8, 2022 at 6:22 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Apr 07, 2022 at 06:09:53PM -0400, Josef Bacik wrote:
> > Just following up on this, I've got hungry kids, I'm about halfway
> > through the new shit.  Depending on how much help kids need with
> > homework I may have this done later tonight, or it'll be tomorrow
> > morning.  Thanks,
>
> It finished after more than 24H, but that didn't seem to be enough, quite.
> Hopefully we're getting closer, though :)
>

Yup we're getting there, I'm about 90% done with the new stuff, and
that'll delete block pointers that we can't find good matches for.

Also it'll cache the results of the disk wide search, so it'll have to
do it once per root instead of for every bad block we find, which will
drastically cut down on the runtime.  I'll let you know when it's
ready, figure another hour or so.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-08 10:23                                                                                                                                                                                             ` Josef Bacik
@ 2022-04-08 20:09                                                                                                                                                                                               ` Josef Bacik
  2022-04-11  1:37                                                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-08 20:09 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Martin Steigerwald, linux-btrfs

On Fri, Apr 8, 2022 at 6:23 AM Josef Bacik <josef@toxicpanda.com> wrote:
>
> On Fri, Apr 8, 2022 at 6:22 AM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Thu, Apr 07, 2022 at 06:09:53PM -0400, Josef Bacik wrote:
> > > Just following up on this, I've got hungry kids, I'm about halfway
> > > through the new shit.  Depending on how much help kids need with
> > > homework I may have this done later tonight, or it'll be tomorrow
> > > morning.  Thanks,
> >
> > It finished after more than 24H, but that didn't seem to be enough, quite.
> > Hopefully we're getting closer, though :)
> >
>
> Yup we're getting there, I'm about 90% done with the new stuff, and
> that'll delete block pointers that we can't find good matches for.
>
> Also it'll cache the results of the disk wide search, so it'll have to
> do it once per root instead of for every bad block we find, which will
> drastically cut down on the runtime.  I'll let you know when it's
> ready, figure another hour or so.  Thanks,
>

Course the last 10% takes the longest, but I corrupted a local file
system and ran it to shake out all the stupid bugs.  Go ahead and pull
and run

./btrfs rescue tree-recover /dev/whatever

and then *hopefully* you can just run btrfs check --repair, but it may
fail out with a "btrfs unable to find ref byte", which is what I was
seeing locally.  I'm fixing that but it's tricky and may be a while.
If you hit that then go ahead and use --init-extent-tree
--init-csum-tree and let it ride.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-08 20:09                                                                                                                                                                                               ` Josef Bacik
@ 2022-04-11  1:37                                                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-04-11  1:37 UTC (permalink / raw)
  To: Josef Bacik; +Cc: Martin Steigerwald, linux-btrfs

On Fri, Apr 08, 2022 at 04:09:34PM -0400, Josef Bacik wrote:
> Course the last 10% takes the longest, but I corrupted a local file
> system and ran it to shake out all the stupid bugs.  Go ahead and pull
> and run
> 
> ./btrfs rescue tree-recover /dev/whatever
> 
> and then *hopefully* you can just run btrfs check --repair, but it may
> fail out with a "btrfs unable to find ref byte", which is what I was
> seeing locally.  I'm fixing that but it's tricky and may be a while.
> If you hit that then go ahead and use --init-extent-tree
> --init-csum-tree and let it ride.  Thanks,

Sorry for the delay, too many things this weekend and I didn't do Email.

gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -5
commit 580e1b4db52a0ec98661bec7310450d84ea1f441
Author: Josef Bacik <josef@toxicpanda.com>
Date:   Fri Apr 8 16:07:48 2022 -0400

    handle fstree's differently

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
Ignoring transid failure
FS_INFO IS 0x558174d9cbc0
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Couldn't find the last root for 8
ERROR: failed to read block groups: Input/output error
FS_INFO AFTER IS 0x558174d9cbc0
deleting slot 94 in block 21069824
FS_INFO IS 0x558174e1f430
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Couldn't find the last root for 8
ERROR: failed to read block groups: Input/output error
FS_INFO AFTER IS 0x558174e1f430
Found completely clean tree for 3 in backup root, replacing
ERROR: Still have update after a repair loop, bailing
Tree recover failed


Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-06 19:13                                                                                                                                       ` Marc MERLIN
  2022-04-06 19:45                                                                                                                                         ` Zygo Blaxell
  2022-04-06 20:38                                                                                                                                         ` figuring out why transient double raid failure caused a fair amount of btrfs corruption Marc MERLIN
@ 2022-04-22 18:48                                                                                                                                         ` Marc MERLIN
  2022-04-22 19:46                                                                                                                                           ` Josef Bacik
  2 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-22 18:48 UTC (permalink / raw)
  To: Zygo Blaxell; +Cc: Josef Bacik, linux-btrfs

Back on list, Josef made a lot of changes to btrfs-progs for me (thanks
Josef)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x55c6d219bbc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55c6d219bbc0
Checking root 2
Checking root 4
Checking root 5
Checking root 7
Checking root 9
Checking root 11221
Checking root 11222
Checking root 11223

After that ./btrfs check --init-extent-tree /dev/mapper/dshelf1 has been
running more for than 48H now

Starting repair.
Opening filesystem to check...
checksum verify failed on 15645878108160 found 00000027 wanted 0000001B
checksum verify failed on 11821979287552 found 000000A9 wanted FFFFFFFB
checksum verify failed on 11822142046208 found 000000AC wanted FFFFFFC4
(...)
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Failed to find [13576823668736, 168, 16384]
btrfs unable to find ref byte nr 13577801809920 parent 0 root 1  owner 0 offset 0
path->slots[0]: 134 path->nodes[0]:
leaf 49709056 items 169 free space 7975 generation 1602090 owner EXTENT_TREE
leaf 49709056 flags 0x0() backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
        item 0 key (13400696422400 BLOCK_GROUP_ITEM 1073741824) itemoff 16259 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 1 key (13401770164224 BLOCK_GROUP_ITEM 1073741824) itemoff 16235 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
(...)
Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent
Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent
Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk.
Device extent[1, 11595375706112, 1073741824] didn't find the relative chunk.
Device extent[1, 11596449447936, 1073741824] didn't find the relative chunk.
Device extent[1, 11597523189760, 1073741824] didn't find the relative chunk.
Device extent[1, 11598596931584, 1073741824] didn't find the relative chunk.
(...)
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
ref mismatch on [12582912 4096] extent item 0, found 1
data backref 12582912 root 11223 owner 260 offset 131072 num_refs 0 not found in extent tree
incorrect local backref count on 12582912 root 11223 owner 260 offset 131072 found 1 wanted 0 back 0x5604b40697a0
backpointer mismatch on [12582912 4096]
adding new data backref on 12582912 root 11223 owner 260 offset 131072 found 1
Repaired extent references for 12582912
ref mismatch on [12587008 4096] extent item 0, found 1
data backref 12587008 root 11223 owner 261 offset 20480 num_refs 0 not found in extent tree
incorrect local backref count on 12587008 root 11223 owner 261 offset 20480 found 1 wanted 0 back 0x5604b4069a00
backpointer mismatch on [12587008 4096]
adding new data backref on 12587008 root 11223 owner 261 offset 20480 found 1
Repaired extent references for 12587008
ref mismatch on [12591104 4096] extent item 0, found 1
(...)
incorrect local backref count on 20963328 parent 33718272 owner 0 offset 0 found 1 wanted 0 back 0x56043bdf6d80
backpointer mismatch on [20963328 4096]
adding new data backref on 20963328 parent 33718272 owner 0 offset 0 found 1
Repaired extent references for 20963328
ref mismatch on [20967424 4096] extent item 0, found 1
data backref 20967424 parent 33718272 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 20967424 parent 33718272 owner 0 offset 0 found 1 wanted 0 back 0x56043bdf6fe0
backpointer mismatch on [20967424 4096]
adding new data backref on 20967424 parent 33718272 owner 0 offset 0 found 1
Repaired extent references for 20967424
ref mismatch on [20971520 16384] extent item 0, found 1
tree backref 20971520 root 3 not found in extent tree
backpointer mismatch on [20971520 16384]
adding new tree backref on start 20971520 len 16384 parent 0 root 3
Repaired extent references for 20971520
ref mismatch on [20987904 16384] extent item 0, found 1
tree backref 20987904 root 3 not found in extent tree
backpointer mismatch on [20987904 16384]
adding new tree backref on start 20987904 len 16384 parent 0 root 3
Repaired extent references for 20987904
ref mismatch on [21004288 16384] extent item 0, found 1
tree backref 21004288 root 3 not found in extent tree
(..)
adding new tree backref on start 44531712 len 16384 parent 50872320 root 50872320
Repaired extent references for 44531712
ref mismatch on [44548096 16384] extent item 0, found 1
tree backref 44548096 parent 47398912 not found in extent tree
backpointer mismatch on [44548096 16384]
adding new tree backref on start 44548096 len 16384 parent 47398912 root 47398912
Repaired extent references for 44548096
ref mismatch on [44564480 16384] extent item 0, found 1
tree backref 44564480 root 7 not found in extent tree
backpointer mismatch on [44564480 16384]
adding new tree backref on start 44564480 len 16384 parent 0 root 7
Repaired extent references for 44564480
ref mismatch on [44580864 16384] extent item 0, found 1
tree backref 44580864 root 7 not found in extent tree
backpointer mismatch on [44580864 16384]
adding new tree backref on start 44580864 len 16384 parent 0 root 7
Repaired extent references for 44580864
ref mismatch on [44597248 16384] extent item 0, found 1
tree backref 44597248 root 7 not found in extent tree
backpointer mismatch on [44597248 16384]
adding new tree backref on start 44597248 len 16384 parent 0 root 7
Repaired extent references for 44597248
ref mismatch on [44613632 16384] extent item 0, found 1
tree backref 44613632 parent 47398912 not found in extent tree
backpointer mismatch on [44613632 16384]
adding new tree backref on start 44613632 len 16384 parent 47398912 root 47398912
Repaired extent references for 44613632
ref mismatch on [44630016 16384] extent item 0, found 1
tree backref 44630016 parent 47398912 not found in extent tree
backpointer mismatch on [44630016 16384]
adding new tree backref on start 44630016 len 16384 parent 47398912 root 47398912
(...)

gargamel:/var/local/src# grep -c 'adding new tree backref on start' checkrepair1 
9026

It's been running for close to 3 days, and I'm a bit confused that it's repairing so
many things when it wast just a minute's worth of potential corruption.

Do I keep running for many more days to see where this goes, or at this point 
73,860 lines of output is not a bad sign and I should look at restoring from backup?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-22 18:48                                                                                                                                         ` Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) Marc MERLIN
@ 2022-04-22 19:46                                                                                                                                           ` Josef Bacik
  2022-04-22 20:01                                                                                                                                             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-22 19:46 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Zygo Blaxell, linux-btrfs

On Fri, Apr 22, 2022 at 2:48 PM Marc MERLIN <marc@merlins.org> wrote:
>
> Back on list, Josef made a lot of changes to btrfs-progs for me (thanks
> Josef)
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> FS_INFO IS 0x55c6d219bbc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x55c6d219bbc0
> Checking root 2
> Checking root 4
> Checking root 5
> Checking root 7
> Checking root 9
> Checking root 11221
> Checking root 11222
> Checking root 11223
>
> After that ./btrfs check --init-extent-tree /dev/mapper/dshelf1 has been
> running more for than 48H now
>
> Starting repair.
> Opening filesystem to check...
> checksum verify failed on 15645878108160 found 00000027 wanted 0000001B
> checksum verify failed on 11821979287552 found 000000A9 wanted FFFFFFFB
> checksum verify failed on 11822142046208 found 000000AC wanted FFFFFFC4
> (...)
> checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
> checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
> Failed to find [13576823668736, 168, 16384]
> btrfs unable to find ref byte nr 13577801809920 parent 0 root 1  owner 0 offset 0
> path->slots[0]: 134 path->nodes[0]:
> leaf 49709056 items 169 free space 7975 generation 1602090 owner EXTENT_TREE
> leaf 49709056 flags 0x0() backref revision 1
> fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
> chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
>         item 0 key (13400696422400 BLOCK_GROUP_ITEM 1073741824) itemoff 16259 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 1 key (13401770164224 BLOCK_GROUP_ITEM 1073741824) itemoff 16235 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
> (...)
> Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent
> Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent
> Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk.
> Device extent[1, 11595375706112, 1073741824] didn't find the relative chunk.
> Device extent[1, 11596449447936, 1073741824] didn't find the relative chunk.
> Device extent[1, 11597523189760, 1073741824] didn't find the relative chunk.
> Device extent[1, 11598596931584, 1073741824] didn't find the relative chunk.
> (...)
> Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
> Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
> ref mismatch on [12582912 4096] extent item 0, found 1
> data backref 12582912 root 11223 owner 260 offset 131072 num_refs 0 not found in extent tree
> incorrect local backref count on 12582912 root 11223 owner 260 offset 131072 found 1 wanted 0 back 0x5604b40697a0
> backpointer mismatch on [12582912 4096]
> adding new data backref on 12582912 root 11223 owner 260 offset 131072 found 1
> Repaired extent references for 12582912
> ref mismatch on [12587008 4096] extent item 0, found 1
> data backref 12587008 root 11223 owner 261 offset 20480 num_refs 0 not found in extent tree
> incorrect local backref count on 12587008 root 11223 owner 261 offset 20480 found 1 wanted 0 back 0x5604b4069a00
> backpointer mismatch on [12587008 4096]
> adding new data backref on 12587008 root 11223 owner 261 offset 20480 found 1
> Repaired extent references for 12587008
> ref mismatch on [12591104 4096] extent item 0, found 1
> (...)
> incorrect local backref count on 20963328 parent 33718272 owner 0 offset 0 found 1 wanted 0 back 0x56043bdf6d80
> backpointer mismatch on [20963328 4096]
> adding new data backref on 20963328 parent 33718272 owner 0 offset 0 found 1
> Repaired extent references for 20963328
> ref mismatch on [20967424 4096] extent item 0, found 1
> data backref 20967424 parent 33718272 owner 0 offset 0 num_refs 0 not found in extent tree
> incorrect local backref count on 20967424 parent 33718272 owner 0 offset 0 found 1 wanted 0 back 0x56043bdf6fe0
> backpointer mismatch on [20967424 4096]
> adding new data backref on 20967424 parent 33718272 owner 0 offset 0 found 1
> Repaired extent references for 20967424
> ref mismatch on [20971520 16384] extent item 0, found 1
> tree backref 20971520 root 3 not found in extent tree
> backpointer mismatch on [20971520 16384]
> adding new tree backref on start 20971520 len 16384 parent 0 root 3
> Repaired extent references for 20971520
> ref mismatch on [20987904 16384] extent item 0, found 1
> tree backref 20987904 root 3 not found in extent tree
> backpointer mismatch on [20987904 16384]
> adding new tree backref on start 20987904 len 16384 parent 0 root 3
> Repaired extent references for 20987904
> ref mismatch on [21004288 16384] extent item 0, found 1
> tree backref 21004288 root 3 not found in extent tree
> (..)
> adding new tree backref on start 44531712 len 16384 parent 50872320 root 50872320
> Repaired extent references for 44531712
> ref mismatch on [44548096 16384] extent item 0, found 1
> tree backref 44548096 parent 47398912 not found in extent tree
> backpointer mismatch on [44548096 16384]
> adding new tree backref on start 44548096 len 16384 parent 47398912 root 47398912
> Repaired extent references for 44548096
> ref mismatch on [44564480 16384] extent item 0, found 1
> tree backref 44564480 root 7 not found in extent tree
> backpointer mismatch on [44564480 16384]
> adding new tree backref on start 44564480 len 16384 parent 0 root 7
> Repaired extent references for 44564480
> ref mismatch on [44580864 16384] extent item 0, found 1
> tree backref 44580864 root 7 not found in extent tree
> backpointer mismatch on [44580864 16384]
> adding new tree backref on start 44580864 len 16384 parent 0 root 7
> Repaired extent references for 44580864
> ref mismatch on [44597248 16384] extent item 0, found 1
> tree backref 44597248 root 7 not found in extent tree
> backpointer mismatch on [44597248 16384]
> adding new tree backref on start 44597248 len 16384 parent 0 root 7
> Repaired extent references for 44597248
> ref mismatch on [44613632 16384] extent item 0, found 1
> tree backref 44613632 parent 47398912 not found in extent tree
> backpointer mismatch on [44613632 16384]
> adding new tree backref on start 44613632 len 16384 parent 47398912 root 47398912
> Repaired extent references for 44613632
> ref mismatch on [44630016 16384] extent item 0, found 1
> tree backref 44630016 parent 47398912 not found in extent tree
> backpointer mismatch on [44630016 16384]
> adding new tree backref on start 44630016 len 16384 parent 47398912 root 47398912
> (...)
>
> gargamel:/var/local/src# grep -c 'adding new tree backref on start' checkrepair1
> 9026
>
> It's been running for close to 3 days, and I'm a bit confused that it's repairing so
> many things when it wast just a minute's worth of potential corruption.
>
> Do I keep running for many more days to see where this goes, or at this point
> 73,860 lines of output is not a bad sign and I should look at restoring from backup?
>

We're doing the --init-extent-tree thing, which deletes the whole
extent tree and rebuilds it.  This isn't the fast path, 3 days is
super shitty tho.  I don't want to stop it and try and make it faster
because once it's done you should just be golden.  It looks scary, but
that's because it just clears the extent tree and then lets the normal
repair thing do its thing, so it yells loudly about all the messed up
references and then fixes it.

Now if we get to Monday and it's still running I can take a crack at
making it faster.  I was hoping it would only take a day or two, but
we're balancing me trying to make it better and possibly fucking it up
with letting it take the rest of our lives but be correct.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-22 19:46                                                                                                                                           ` Josef Bacik
@ 2022-04-22 20:01                                                                                                                                             ` Marc MERLIN
  2022-04-23 20:12                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-22 20:01 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Fri, Apr 22, 2022 at 03:46:08PM -0400, Josef Bacik wrote:
> We're doing the --init-extent-tree thing, which deletes the whole
> extent tree and rebuilds it.  This isn't the fast path, 3 days is
> super shitty tho.  I don't want to stop it and try and make it faster
> because once it's done you should just be golden.  It looks scary, but
> that's because it just clears the extent tree and then lets the normal
> repair thing do its thing, so it yells loudly about all the messed up
> references and then fixes it.
 
Got it.

> Now if we get to Monday and it's still running I can take a crack at
> making it faster.  I was hoping it would only take a day or two, but
> we're balancing me trying to make it better and possibly fucking it up
> with letting it take the rest of our lives but be correct.  Thanks,

Makes sense. I don't need faster, and it may not be able to go faster
anyway, it's a lot of data. Just wanted to make sure the output and
relative slow results were expected. 
And I'll go with slower but likely to work vs faster but very
experimental :)

That said, as Ted T'so said once, "anything with a progress bar is
faster".
So if it's possible in the future to add some completion percent, even
if approximate, that will help when it can take multiple days (or week+)
to run.

Happy to wait in the meantime, thanks.
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-22 20:01                                                                                                                                             ` Marc MERLIN
@ 2022-04-23 20:12                                                                                                                                               ` Marc MERLIN
  2022-04-23 20:53                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-23 20:12 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Fri, Apr 22, 2022 at 01:01:15PM -0700, Marc MERLIN wrote:
> > Now if we get to Monday and it's still running I can take a crack at
> > making it faster.  I was hoping it would only take a day or two, but
> > we're balancing me trying to make it better and possibly fucking it up
> > with letting it take the rest of our lives but be correct.  Thanks,
> 
> Makes sense. I don't need faster, and it may not be able to go faster
> anyway, it's a lot of data. Just wanted to make sure the output and
> relative slow results were expected. 

Looking at the output, is there any way I can figure out if it's at 5%
or 80% completion?

tree backref 238026752 parent 236814336 not found in extent tree
backpointer mismatch on [238026752 16384]
adding new tree backref on start 238026752 len 16384 parent 236814336 root 236814336
Repaired extent references for 238026752
ref mismatch on [238043136 16384] extent item 0, found 1
tree backref 238043136 parent 236814336 not found in extent tree
backpointer mismatch on [238043136 16384]
adding new tree backref on start 238043136 len 16384 parent 236814336 root 236814336
Repaired extent references for 238043136
ref mismatch on [238059520 16384] extent item 0, found 1
tree backref 238059520 parent 236814336 not found in extent tree
backpointer mismatch on [238059520 16384]
adding new tree backref on start 238059520 len 16384 parent 236814336 root 236814336

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-23 20:12                                                                                                                                               ` Marc MERLIN
@ 2022-04-23 20:53                                                                                                                                                 ` Josef Bacik
  2022-04-24 16:20                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-23 20:53 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sat, Apr 23, 2022 at 4:12 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Fri, Apr 22, 2022 at 01:01:15PM -0700, Marc MERLIN wrote:
> > > Now if we get to Monday and it's still running I can take a crack at
> > > making it faster.  I was hoping it would only take a day or two, but
> > > we're balancing me trying to make it better and possibly fucking it up
> > > with letting it take the rest of our lives but be correct.  Thanks,
> >
> > Makes sense. I don't need faster, and it may not be able to go faster
> > anyway, it's a lot of data. Just wanted to make sure the output and
> > relative slow results were expected.
>
> Looking at the output, is there any way I can figure out if it's at 5%
> or 80% completion?
>
> tree backref 238026752 parent 236814336 not found in extent tree
> backpointer mismatch on [238026752 16384]
> adding new tree backref on start 238026752 len 16384 parent 236814336 root 236814336
> Repaired extent references for 238026752
> ref mismatch on [238043136 16384] extent item 0, found 1
> tree backref 238043136 parent 236814336 not found in extent tree
> backpointer mismatch on [238043136 16384]
> adding new tree backref on start 238043136 len 16384 parent 236814336 root 236814336
> Repaired extent references for 238043136
> ref mismatch on [238059520 16384] extent item 0, found 1
> tree backref 238059520 parent 236814336 not found in extent tree
> backpointer mismatch on [238059520 16384]
> adding new tree backref on start 238059520 len 16384 parent 236814336 root 236814336
>

Hmm I don't know, that's byte 227, but you're in fs trees now, so
hopefully soon?  I've got some free time, let me rewrite this to be
less stupid and see if I can get it done before your thing finishes,
and I'll add some sort of progress indicator.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-23 20:53                                                                                                                                                 ` Josef Bacik
@ 2022-04-24 16:20                                                                                                                                                   ` Josef Bacik
  2022-04-24 16:24                                                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-24 16:20 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sat, Apr 23, 2022 at 4:53 PM Josef Bacik <josef@toxicpanda.com> wrote:
>
> On Sat, Apr 23, 2022 at 4:12 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Fri, Apr 22, 2022 at 01:01:15PM -0700, Marc MERLIN wrote:
> > > > Now if we get to Monday and it's still running I can take a crack at
> > > > making it faster.  I was hoping it would only take a day or two, but
> > > > we're balancing me trying to make it better and possibly fucking it up
> > > > with letting it take the rest of our lives but be correct.  Thanks,
> > >
> > > Makes sense. I don't need faster, and it may not be able to go faster
> > > anyway, it's a lot of data. Just wanted to make sure the output and
> > > relative slow results were expected.
> >
> > Looking at the output, is there any way I can figure out if it's at 5%
> > or 80% completion?
> >
> > tree backref 238026752 parent 236814336 not found in extent tree
> > backpointer mismatch on [238026752 16384]
> > adding new tree backref on start 238026752 len 16384 parent 236814336 root 236814336
> > Repaired extent references for 238026752
> > ref mismatch on [238043136 16384] extent item 0, found 1
> > tree backref 238043136 parent 236814336 not found in extent tree
> > backpointer mismatch on [238043136 16384]
> > adding new tree backref on start 238043136 len 16384 parent 236814336 root 236814336
> > Repaired extent references for 238043136
> > ref mismatch on [238059520 16384] extent item 0, found 1
> > tree backref 238059520 parent 236814336 not found in extent tree
> > backpointer mismatch on [238059520 16384]
> > adding new tree backref on start 238059520 len 16384 parent 236814336 root 236814336
> >
>
> Hmm I don't know, that's byte 227, but you're in fs trees now, so
> hopefully soon?  I've got some free time, let me rewrite this to be
> less stupid and see if I can get it done before your thing finishes,
> and I'll add some sort of progress indicator.  Thanks,
>

Alright you can kill the command, pull my tree, then run

btrfs rescue init-extent-tree <blah>

This do the straight re-init of the extent tree without all the
looping and extra stuff that fsck does, so hopefully will be faster?
Also it keeps track of how many bytes we've processed, as well as how
many bytes we may have to process, so you'll get a better feel for the
progress.  I have a scratch fs here that has like 30gib of metadata
with 1tib of data and it took around 30 seconds to run.  It has no
snapshots, I ran it on a fsstress generated fs with like 5k snapshots
and it took about a minute, but with significantly less metadata/data.

Once you run this you'll still have to run --repair, but it shouldn't
have to touch the extent tree.  If you see it messing with the extent
tree stop it and let me know what it's complaining about and I'll fix
it up.  I'm pretty sure my thing is going to work, but your fs is more
complex than I can generate on the fly.

Also, the indicator may show "processed" as less than "possible" bytes
when it moves onto the next root, this is normal for snapshots, if
we've already processed a subtree we won't walk down it, so it's fine
if you see that.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-24 16:20                                                                                                                                                   ` Josef Bacik
@ 2022-04-24 16:24                                                                                                                                                     ` Marc MERLIN
  2022-04-24 17:09                                                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-24 16:24 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 12:20:42PM -0400, Josef Bacik wrote:
> On Sat, Apr 23, 2022 at 4:53 PM Josef Bacik <josef@toxicpanda.com> wrote:
> >
> > On Sat, Apr 23, 2022 at 4:12 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Fri, Apr 22, 2022 at 01:01:15PM -0700, Marc MERLIN wrote:
> > > > > Now if we get to Monday and it's still running I can take a crack at
> > > > > making it faster.  I was hoping it would only take a day or two, but
> > > > > we're balancing me trying to make it better and possibly fucking it up
> > > > > with letting it take the rest of our lives but be correct.  Thanks,
> > > >
> > > > Makes sense. I don't need faster, and it may not be able to go faster
> > > > anyway, it's a lot of data. Just wanted to make sure the output and
> > > > relative slow results were expected.
> > >
> > > Looking at the output, is there any way I can figure out if it's at 5%
> > > or 80% completion?
> > >
> > > tree backref 238026752 parent 236814336 not found in extent tree
> > > backpointer mismatch on [238026752 16384]
> > > adding new tree backref on start 238026752 len 16384 parent 236814336 root 236814336
> > > Repaired extent references for 238026752
> > > ref mismatch on [238043136 16384] extent item 0, found 1
> > > tree backref 238043136 parent 236814336 not found in extent tree
> > > backpointer mismatch on [238043136 16384]
> > > adding new tree backref on start 238043136 len 16384 parent 236814336 root 236814336
> > > Repaired extent references for 238043136
> > > ref mismatch on [238059520 16384] extent item 0, found 1
> > > tree backref 238059520 parent 236814336 not found in extent tree
> > > backpointer mismatch on [238059520 16384]
> > > adding new tree backref on start 238059520 len 16384 parent 236814336 root 236814336
> > >
> >
> > Hmm I don't know, that's byte 227, but you're in fs trees now, so
> > hopefully soon?  I've got some free time, let me rewrite this to be
> > less stupid and see if I can get it done before your thing finishes,
> > and I'll add some sort of progress indicator.  Thanks,
> >
> 
> Alright you can kill the command, pull my tree, then run
> 
> btrfs rescue init-extent-tree <blah>
 
Thanks. Tried the new version
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x55bf50e98bc0
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55bf50e98bc0
Walking all our trees and pinning down the currently accessible blocks
checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d
checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188
checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858
checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82
(...)
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found
0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found
0x46985a93
Clearing the extent root and re-init'ing the block groups
processed 1556480 of 0 possible bytes
parent transid verify failed on 58736640 wanted 1619007 found 1619009
processed 16384 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
processed 32768 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58753024 wanted 1619007 found 1619009
parent transid verify failed on 58753024 wanted 1619007 found 1619009
processed 49152 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
corrupt leaf: root=2 block=58736640 physical=1140867072 slot=107, bad
key order, prev (169143697408 192 1073741824) current (12582912 192
8388608)
kernel-shared/extent-tree.c:1287: btrfs_inc_extent_ref: BUG_ON `err`
triggered, value -1
./btrfs(+0x29683)[0x55bf4fea1683]
./btrfs(btrfs_inc_extent_ref+0x522)[0x55bf4fea3061]
./btrfs(+0x8ba18)[0x55bf4ff03a18]
./btrfs(+0x8bb27)[0x55bf4ff03b27]
./btrfs(+0x8bd8f)[0x55bf4ff03d8f]
./btrfs(btrfs_init_extent_tree+0x226)[0x55bf4ff04092]
./btrfs(+0x83931)[0x55bf4fefb931]
./btrfs(handle_command_group+0x49)[0x55bf4fe9017b]
./btrfs(main+0x94)[0x55bf4fe90275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7faebe73a7fd]
./btrfs(_start+0x2a)[0x55bf4fe8fe1a]
Aborted

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-24 16:24                                                                                                                                                     ` Marc MERLIN
@ 2022-04-24 17:09                                                                                                                                                       ` Josef Bacik
  2022-04-24 18:43                                                                                                                                                         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-24 17:09 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 12:24 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, Apr 24, 2022 at 12:20:42PM -0400, Josef Bacik wrote:
> > On Sat, Apr 23, 2022 at 4:53 PM Josef Bacik <josef@toxicpanda.com> wrote:
> > >
> > > On Sat, Apr 23, 2022 at 4:12 PM Marc MERLIN <marc@merlins.org> wrote:
> > > >
> > > > On Fri, Apr 22, 2022 at 01:01:15PM -0700, Marc MERLIN wrote:
> > > > > > Now if we get to Monday and it's still running I can take a crack at
> > > > > > making it faster.  I was hoping it would only take a day or two, but
> > > > > > we're balancing me trying to make it better and possibly fucking it up
> > > > > > with letting it take the rest of our lives but be correct.  Thanks,
> > > > >
> > > > > Makes sense. I don't need faster, and it may not be able to go faster
> > > > > anyway, it's a lot of data. Just wanted to make sure the output and
> > > > > relative slow results were expected.
> > > >
> > > > Looking at the output, is there any way I can figure out if it's at 5%
> > > > or 80% completion?
> > > >
> > > > tree backref 238026752 parent 236814336 not found in extent tree
> > > > backpointer mismatch on [238026752 16384]
> > > > adding new tree backref on start 238026752 len 16384 parent 236814336 root 236814336
> > > > Repaired extent references for 238026752
> > > > ref mismatch on [238043136 16384] extent item 0, found 1
> > > > tree backref 238043136 parent 236814336 not found in extent tree
> > > > backpointer mismatch on [238043136 16384]
> > > > adding new tree backref on start 238043136 len 16384 parent 236814336 root 236814336
> > > > Repaired extent references for 238043136
> > > > ref mismatch on [238059520 16384] extent item 0, found 1
> > > > tree backref 238059520 parent 236814336 not found in extent tree
> > > > backpointer mismatch on [238059520 16384]
> > > > adding new tree backref on start 238059520 len 16384 parent 236814336 root 236814336
> > > >
> > >
> > > Hmm I don't know, that's byte 227, but you're in fs trees now, so
> > > hopefully soon?  I've got some free time, let me rewrite this to be
> > > less stupid and see if I can get it done before your thing finishes,
> > > and I'll add some sort of progress indicator.  Thanks,
> > >
> >
> > Alright you can kill the command, pull my tree, then run
> >
> > btrfs rescue init-extent-tree <blah>
>
> Thanks. Tried the new version
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
> FS_INFO IS 0x55bf50e98bc0
> checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x55bf50e98bc0
> Walking all our trees and pinning down the currently accessible blocks
> checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d
> checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188
> checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858
> checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82
> (...)
> checksum verify failed on 13577821011968 wanted 0x9a29aff5 found
> 0x2cdff391
> checksum verify failed on 15645781196800 wanted 0xef669b11 found
> 0x46985a93
> Clearing the extent root and re-init'ing the block groups
> processed 1556480 of 0 possible bytes
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> processed 16384 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> processed 32768 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58753024 wanted 1619007 found 1619009
> parent transid verify failed on 58753024 wanted 1619007 found 1619009
> processed 49152 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> parent transid verify failed on 58736640 wanted 1619007 found 1619009
> corrupt leaf: root=2 block=58736640 physical=1140867072 slot=107, bad
> key order, prev (169143697408 192 1073741824) current (12582912 192
> 8388608)
> kernel-shared/extent-tree.c:1287: btrfs_inc_extent_ref: BUG_ON `err`
> triggered, value -1
> ./btrfs(+0x29683)[0x55bf4fea1683]
> ./btrfs(btrfs_inc_extent_ref+0x522)[0x55bf4fea3061]
> ./btrfs(+0x8ba18)[0x55bf4ff03a18]
> ./btrfs(+0x8bb27)[0x55bf4ff03b27]
> ./btrfs(+0x8bd8f)[0x55bf4ff03d8f]
> ./btrfs(btrfs_init_extent_tree+0x226)[0x55bf4ff04092]
> ./btrfs(+0x83931)[0x55bf4fefb931]
> ./btrfs(handle_command_group+0x49)[0x55bf4fe9017b]
> ./btrfs(main+0x94)[0x55bf4fe90275]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7faebe73a7fd]
> ./btrfs(_start+0x2a)[0x55bf4fe8fe1a]
> Aborted

Argh you still have bogus stuff for some reason, maybe it's because of
cancelling the rescue.  Re-run btrfs rescue tree-recover, and do it a
couple of times to make sure it's stopped complaining.  If it does
complain after the first run please let me know, it really should fix
everything the first go around.  If it's not then I need to try and
reproduce that locally and figure out wtf it's doing.  Once
tree-recover runs cleanly you can run btrfs rescue init-extent-tree,
then hopefully you'll be good to go.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-24 17:09                                                                                                                                                       ` Josef Bacik
@ 2022-04-24 18:43                                                                                                                                                         ` Marc MERLIN
  2022-04-24 19:17                                                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-24 18:43 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 01:09:45PM -0400, Josef Bacik wrote:
> Argh you still have bogus stuff for some reason, maybe it's because of
> cancelling the rescue.  Re-run btrfs rescue tree-recover, and do it a
> couple of times to make sure it's stopped complaining.  If it does
> complain after the first run please let me know, it really should fix
> everything the first go around.  If it's not then I need to try and
> reproduce that locally and figure out wtf it's doing.  Once
> tree-recover runs cleanly you can run btrfs rescue init-extent-tree,
> then hopefully you'll be good to go.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x555a360efbc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555a360efbc0
Checking root 2
Checking root 4
Checking root 5
Checking root 7
Checking root 9
Checking root 11221
Checking root 11222
Checking root 11223
Checking root 11224
Checking root 159785
Checking root 159787
Checking root 160494
Checking root 160496
Checking root 161197
Checking root 161199
Checking root 162628
Checking root 162632
Checking root 162645
Checking root 163298
Checking root 163302
Checking root 163303
Checking root 163316
Checking root 163318
Checking root 163916
Checking root 163920
Checking root 163921
Checking root 164620
Checking root 164624
Checking root 164633
Checking root 165098
Checking root 165100
Checking root 165198
Checking root 165200
Checking root 165294
Checking root 165298
Checking root 165299
Checking root 18446744073709551607
Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x56080e544bc0
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
Couldn't find the last root for 8
FS_INFO AFTER IS 0x56080e544bc0
Walking all our trees and pinning down the currently accessible blocks
checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d
checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188
checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858
checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82
checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed
(...)
checksum verify failed on 13577810395136 wanted 0x4171ba0a found 0xda4963ab
checksum verify failed on 46576713728 wanted 0xbb1ee48e found 0x8809f6a6
checksum verify failed on 13577126330368 wanted 0xf7264639 found 0x8287ec94
checksum verify failed on 15645432397824 wanted 0xc706f8b6 found 0xf0425483
checksum verify failed on 15645954080768 wanted 0x3ff10ddf found 0x77ab2f14
checksum verify failed on 15645954129920 wanted 0x085fc24a found 0x941acdde
checksum verify failed on 15645954195456 wanted 0x6c1c72a4 found 0xb06d5dd9
checksum verify failed on 15645262348288 wanted 0x433325c8 found 0x81ea056f
checksum verify failed on 15645608116224 wanted 0x31706547 found 0x314fad1d
checksum verify failed on 15645262413824 wanted 0x3affbd35 found 0x96d53d25
checksum verify failed on 15646053974016 wanted 0x98bd62bc found 0xdaa7dd44
checksum verify failed on 15646006722560 wanted 0x760a06a0 found 0xb44fcd95
checksum verify failed on 15645980491776 wanted 0xe1e674a7 found 0x0fdfda2c
checksum verify failed on 15645526884352 wanted 0xc2d409c1 found 0xb3eee8d2
checksum verify failed on 12511656394752 wanted 0x607d7b9e found 0xec111bf5
checksum verify failed on 13577013936128 wanted 0x4ba00b03 found 0x64614751
checksum verify failed on 13577199878144 wanted 0x6e9e8bc6 found 0x61063457
checksum verify failed on 13577399156736 wanted 0x2869b8c7 found 0xbb1119e1
checksum verify failed on 12512437698560 wanted 0xca43b3f8 found 0xd7f6db69
checksum verify failed on 13577503686656 wanted 0xd81b7702 found 0x95a3c9a6
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Clearing the extent root and re-init'ing the block groups
processed 1556480 of 0 possible bytes
processed 1228800 of 0 possible bytesFailed to find [7750833868800, 168, 262144]
Segmentation fault

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559add0 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559add0 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48
#1  0x000055555559afd4 in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=99696640)
    at common/extent-cache.c:179
#2  0x0000555555584b67 in set_extent_bits (tree=0x10000000060, start=99696640, end=99696639, bits=bits@entry=1)
    at kernel-shared/extent_io.c:380
#3  0x0000555555584d9a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x000055555558577d in set_extent_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/extent_io.c:976
#5  0x000055555557bbaf in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/disk-io.c:2222
#6  0x000055555557ed74 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434045, parent=0, 
    iref=0xffffffffffffffe4, path=0x555555b82ed0, root=<optimized out>) at kernel-shared/extent-tree.c:1084
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434045, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b82ed0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1196
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555558a89680, root=root@entry=0x55555564cde0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0)
    at kernel-shared/extent-tree.c:1261
#9  0x00005555555dfa18 in process_eb (trans=trans@entry=0x555558a89680, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559cfcb20, 
    current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:499
#10 0x00005555555dfb27 in process_eb (trans=trans@entry=0x555558a89680, root=root@entry=0x55555564cde0, eb=0x55555566b790, 
    current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:574
#11 0x00005555555dfd8f in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:645
#12 0x00005555555e0092 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1d0 "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:779
#13 0x00005555555d7931 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#17 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-24 18:43                                                                                                                                                         ` Marc MERLIN
@ 2022-04-24 19:17                                                                                                                                                           ` Josef Bacik
  2022-04-24 19:44                                                                                                                                                             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-24 19:17 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 2:43 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, Apr 24, 2022 at 01:09:45PM -0400, Josef Bacik wrote:
> > Argh you still have bogus stuff for some reason, maybe it's because of
> > cancelling the rescue.  Re-run btrfs rescue tree-recover, and do it a
> > couple of times to make sure it's stopped complaining.  If it does
> > complain after the first run please let me know, it really should fix
> > everything the first go around.  If it's not then I need to try and
> > reproduce that locally and figure out wtf it's doing.  Once
> > tree-recover runs cleanly you can run btrfs rescue init-extent-tree,
> > then hopefully you'll be good to go.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> FS_INFO IS 0x555a360efbc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x555a360efbc0
> Checking root 2
> Checking root 4
> Checking root 5
> Checking root 7
> Checking root 9
> Checking root 11221
> Checking root 11222
> Checking root 11223
> Checking root 11224
> Checking root 159785
> Checking root 159787
> Checking root 160494
> Checking root 160496
> Checking root 161197
> Checking root 161199
> Checking root 162628
> Checking root 162632
> Checking root 162645
> Checking root 163298
> Checking root 163302
> Checking root 163303
> Checking root 163316
> Checking root 163318
> Checking root 163916
> Checking root 163920
> Checking root 163921
> Checking root 164620
> Checking root 164624
> Checking root 164633
> Checking root 165098
> Checking root 165100
> Checking root 165198
> Checking root 165200
> Checking root 165294
> Checking root 165298
> Checking root 165299
> Checking root 18446744073709551607
> Tree recovery finished, you can run check now
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
> FS_INFO IS 0x56080e544bc0
> checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4

Ok this is sort of maddening, IDK how we'll trip over this broken
block here, but not with tree-recover.  Can you repull, build, and
then run init-extent-tree.  As soon as you see the JOSEF message you
can kill it and send me the output, I need to figure out wtf is going
on here.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-24 19:17                                                                                                                                                           ` Josef Bacik
@ 2022-04-24 19:44                                                                                                                                                             ` Marc MERLIN
  2022-04-24 20:01                                                                                                                                                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-24 19:44 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 03:17:33PM -0400, Josef Bacik wrote:
> Ok this is sort of maddening, IDK how we'll trip over this broken
> block here, but not with tree-recover.  Can you repull, build, and
> then run init-extent-tree.  As soon as you see the JOSEF message you
> can kill it and send me the output, I need to figure out wtf is going
> on here.  Thanks,

Mmmh, not sure I got that far.

gargamel:/var/local/src/btrfs-progs-josefbacik# grep -r JOSEF *
Binary file btrfs matches
Binary file btrfsck matches
Binary file btrfs-convert matches
Binary file btrfs-corrupt-block matches
Binary file btrfs-find-root matches
Binary file btrfs-image matches
Binary file btrfs-map-logical matches
Binary file btrfs-select-super matches
Binary file btrfstune matches
kernel-shared/ctree.c:          printf("JOSEF: read node slot on root %llu\n", root->root_key.objectid);
Binary file kernel-shared/ctree.o matches
kernel-shared/disk-io.c:                printf("JOSEF: root %llu\n", root->root_key.objectid);
Binary file kernel-shared/disk-io.o matches
Binary file mkfs.btrfs matches


(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
Couldn't find the last root for 8
checksum verify failed on 58720256 wanted 0x89f5cb13 found 0x04196d22
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks
checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d
checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188
checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858
(...)
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Clearing the extent root and re-init'ing the block groups
processed 1556480 of 0 possible bytes
processed 1228800 of 0 possible bytesFailed to find [7750833868800, 168, 262144]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559ae58 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559ae58 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48
#1  0x000055555559b05c in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=99696640)
    at common/extent-cache.c:179
#2  0x0000555555584bea in set_extent_bits (tree=0x10000000060, start=99696640, end=99696639, bits=bits@entry=1)
    at kernel-shared/extent_io.c:380
#3  0x0000555555584e1d in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x0000555555585800 in set_extent_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/extent_io.c:976
#5  0x000055555557bbf3 in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/disk-io.c:2224
#6  0x000055555557edf7 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434176, parent=0, 
    iref=0xffffffffffffffe4, path=0x555555b83090, root=<optimized out>) at kernel-shared/extent-tree.c:1084
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434176, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b83090, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1196
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0)
    at kernel-shared/extent-tree.c:1261
#9  0x00005555555dfaad in process_eb (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559cfcfa0, 
    current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:499
#10 0x00005555555dfbbc in process_eb (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, eb=0x55555566b790, 
    current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:574
#11 0x00005555555dfe24 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:645
#12 0x00005555555e0127 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1d0 "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:779
#13 0x00005555555d79c6 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#17 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-24 19:44                                                                                                                                                             ` Marc MERLIN
@ 2022-04-24 20:01                                                                                                                                                               ` Josef Bacik
  2022-04-24 20:31                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-24 20:01 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 3:44 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, Apr 24, 2022 at 03:17:33PM -0400, Josef Bacik wrote:
> > Ok this is sort of maddening, IDK how we'll trip over this broken
> > block here, but not with tree-recover.  Can you repull, build, and
> > then run init-extent-tree.  As soon as you see the JOSEF message you
> > can kill it and send me the output, I need to figure out wtf is going
> > on here.  Thanks,
>
> Mmmh, not sure I got that far.
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# grep -r JOSEF *
> Binary file btrfs matches
> Binary file btrfsck matches
> Binary file btrfs-convert matches
> Binary file btrfs-corrupt-block matches
> Binary file btrfs-find-root matches
> Binary file btrfs-image matches
> Binary file btrfs-map-logical matches
> Binary file btrfs-select-super matches
> Binary file btrfstune matches
> kernel-shared/ctree.c:          printf("JOSEF: read node slot on root %llu\n", root->root_key.objectid);
> Binary file kernel-shared/ctree.o matches
> kernel-shared/disk-io.c:                printf("JOSEF: root %llu\n", root->root_key.objectid);
> Binary file kernel-shared/disk-io.o matches
> Binary file mkfs.btrfs matches
>
>
> (gdb) run rescue init-extent-tree /dev/mapper/dshelf1
> Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> FS_INFO IS 0x55555564cbc0
> JOSEF: root 9

Huh ok, it's the UUID tree, weird.  I pushed, can you re-run
tree-recover, you can stop it after it does root 9, I just want to see
what bytenr it thinks the root node is at.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-24 20:01                                                                                                                                                               ` Josef Bacik
@ 2022-04-24 20:31                                                                                                                                                                 ` Marc MERLIN
  2022-04-24 20:32                                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-24 20:31 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 04:01:34PM -0400, Josef Bacik wrote:
> > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1
> > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > FS_INFO IS 0x55555564cbc0
> > JOSEF: root 9
> 
> Huh ok, it's the UUID tree, weird.  I pushed, can you re-run
> tree-recover, you can stop it after it does root 9, I just want to see
> what bytenr it thinks the root node is at.  Thanks,

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks
checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d
checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188
checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858
checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82
checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed
checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c
checksum verify failed on 15645178363904 wanted 0xba219c04 found 0x86a9d7f0
checksum verify failed on 15645917708288 wanted 0x4dab2011 found 0xb6299718
checksum verify failed on 11970896232448 wanted 0x8b800530 found 0x3790ebce
checksum verify failed on 8733701898240 wanted 0x80eed7c5 found 0xd6af2972
(..)
checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Clearing the extent root and re-init'ing the block groups
processed 1556480 of 0 possible bytes
processed 1228800 of 0 possible bytesFailed to find [7750833868800, 168, 262144]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559ae58 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559ae58 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48
#1  0x000055555559b05c in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=99696640)
    at common/extent-cache.c:179
#2  0x0000555555584bea in set_extent_bits (tree=0x10000000060, start=99696640, end=99696639, bits=bits@entry=1)
    at kernel-shared/extent_io.c:380
#3  0x0000555555584e1d in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x0000555555585800 in set_extent_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/extent_io.c:976
#5  0x000055555557bbf3 in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/disk-io.c:2224
#6  0x000055555557edf7 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434176, parent=0, 
    iref=0xffffffffffffffe4, path=0x555555b928c0, root=<optimized out>) at kernel-shared/extent-tree.c:1084
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434176, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b928c0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1196
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0)
    at kernel-shared/extent-tree.c:1261
#9  0x00005555555dfb31 in process_eb (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559cfcbe0, 
    current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:499
#10 0x00005555555dfc40 in process_eb (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, eb=0x55555566b790, 
    current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:574
#11 0x00005555555dfea8 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:645
#12 0x00005555555e01ab in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1d0 "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:779
#13 0x00005555555d79c6 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#17 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-24 20:31                                                                                                                                                                 ` Marc MERLIN
@ 2022-04-24 20:32                                                                                                                                                                   ` Josef Bacik
  2022-04-24 20:54                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-24 20:32 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 4:31 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, Apr 24, 2022 at 04:01:34PM -0400, Josef Bacik wrote:
> > > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1
> > > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
> > > [Thread debugging using libthread_db enabled]
> > > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > > FS_INFO IS 0x55555564cbc0
> > > JOSEF: root 9
> >
> > Huh ok, it's the UUID tree, weird.  I pushed, can you re-run
> > tree-recover, you can stop it after it does root 9, I just want to see
> > what bytenr it thinks the root node is at.  Thanks,
>
> (gdb) run rescue init-extent-tree /dev/mapper/dshelf1
> Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1

Sorry, I need tree-recover, not init-extent-tree.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-24 20:32                                                                                                                                                                   ` Josef Bacik
@ 2022-04-24 20:54                                                                                                                                                                     ` Marc MERLIN
  2022-04-24 21:01                                                                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-24 20:54 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 04:32:23PM -0400, Josef Bacik wrote:
> On Sun, Apr 24, 2022 at 4:31 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sun, Apr 24, 2022 at 04:01:34PM -0400, Josef Bacik wrote:
> > > > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1
> > > > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
> > > > [Thread debugging using libthread_db enabled]
> > > > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > > > FS_INFO IS 0x55555564cbc0
> > > > JOSEF: root 9
> > >
> > > Huh ok, it's the UUID tree, weird.  I pushed, can you re-run
> > > tree-recover, you can stop it after it does root 9, I just want to see
> > > what bytenr it thinks the root node is at.  Thanks,
> >
> > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1
> > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
> 
> Sorry, I need tree-recover, not init-extent-tree.  Thanks,

gdb) run rescue tree-recover /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue
tree-recover /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55555564cbc0
Checking root 2 bytenr 67387392
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 13577819963392
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736
Checking root 11222 bytenr 15645261905920
Checking root 11223 bytenr 13576862547968
Checking root 11224 bytenr 13577126182912
Checking root 159785 bytenr 6781490577408
Checking root 159787 bytenr 15645908385792
Checking root 160494 bytenr 6781491265536
Checking root 160496 bytenr 11822309965824
Checking root 161197 bytenr 6781492101120
Checking root 161199 bytenr 13576850833408
Checking root 162628 bytenr 15645764812800
Checking root 162632 bytenr 6781492756480
Checking root 162645 bytenr 5809981095936
Checking root 163298 bytenr 15645124263936
Checking root 163302 bytenr 6781495197696
Checking root 163303 bytenr 15645365993472
Checking root 163316 bytenr 6781496393728
Checking root 163318 bytenr 15645980491776
Checking root 163916 bytenr 11822437826560
Checking root 163920 bytenr 11971021275136
Checking root 163921 bytenr 11971073802240
Checking root 164620 bytenr 15645434036224
Checking root 164624 bytenr 15645502210048
Checking root 164633 bytenr 15645526884352
Checking root 165098 bytenr 11970667446272
Checking root 165100 bytenr 11970733621248
Checking root 165198 bytenr 12511656394752
Checking root 165200 bytenr 12511677972480
Checking root 165294 bytenr 13576901328896
Checking root 165298 bytenr 13577133326336
Checking root 165299 bytenr 13577191505920
Checking root 18446744073709551607 bytenr 13576823685120
Tree recovery finished, you can run check now
[Inferior 1 (process 6147) exited normally]
(gdb) 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-24 20:54                                                                                                                                                                     ` Marc MERLIN
@ 2022-04-24 21:01                                                                                                                                                                       ` Josef Bacik
       [not found]                                                                                                                                                                         ` <20220424210732.GC29107@merlins.org>
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-24 21:01 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 4:54 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, Apr 24, 2022 at 04:32:23PM -0400, Josef Bacik wrote:
> > On Sun, Apr 24, 2022 at 4:31 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sun, Apr 24, 2022 at 04:01:34PM -0400, Josef Bacik wrote:
> > > > > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1
> > > > > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
> > > > > [Thread debugging using libthread_db enabled]
> > > > > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > > > > FS_INFO IS 0x55555564cbc0
> > > > > JOSEF: root 9
> > > >
> > > > Huh ok, it's the UUID tree, weird.  I pushed, can you re-run
> > > > tree-recover, you can stop it after it does root 9, I just want to see
> > > > what bytenr it thinks the root node is at.  Thanks,
> > >
> > > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1
> > > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
> >
> > Sorry, I need tree-recover, not init-extent-tree.  Thanks,
>
> gdb) run rescue tree-recover /dev/mapper/dshelf1
> Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue
> tree-recover /dev/mapper/dshelf1

Wtf, it's reading the right bytenr, but it's not failing here but
failing when we do the init-extent-tree.  I've pushed something again
to force reads, maybe that's the problem, can you run tree-recover and
then init-extent-tree again?  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
       [not found]                                                                                                                                                                               ` <CAEzrpqcBvh0MC6WeXQ+-80igZhg6t68OcgZnKi6xu+r=njifeA@mail.gmail.com>
@ 2022-04-24 22:38                                                                                                                                                                                 ` Marc MERLIN
  2022-04-24 22:56                                                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-24 22:38 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 05:26:11PM -0400, Josef Bacik wrote:
> On Sun, Apr 24, 2022 at 5:20 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sun, Apr 24, 2022 at 05:13:22PM -0400, Josef Bacik wrote:
> > > On Sun, Apr 24, 2022 at 5:07 PM Marc MERLIN <marc@merlins.org> wrote:
> > > >
> > > > On Sun, Apr 24, 2022 at 05:01:38PM -0400, Josef Bacik wrote:
> > > > > Wtf, it's reading the right bytenr, but it's not failing here but
> > > > > failing when we do the init-extent-tree.  I've pushed something again
> > > > > to force reads, maybe that's the problem, can you run tree-recover and
> > > > > then init-extent-tree again?  Thanks,
> > > >
> > > > off-list:
> > > > Seeing this, that kind of worried me so I stopped it. Is it ok to
> > > > continue?
> > >
> > > Sorry, my printf was wrong, pushed.  Thanks,
> >
> > Better?
> > Do I worry about those new "had to be read from a different mirror" ?
> >
> 
> Ooooh ok, the other mirror is bad, but it finds the right thing.  Ok
> that makes sense.  I pushed another patch just to make sure.  Once
> this is all done you'll need to run a scrub to fix the mirrors but
> this is fine.  Thanks,

No more crash, but:
ERROR: Error adding block group
ERROR: commit_root already set when starting transaction


(gdb) run rescue tree-recover /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue tree-recover /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
block 15645878108160 had to be read from a different mirror, ret 0
Couldn't find the last root for 8
block 58720256 had to be read from a different mirror, ret 0
block 58720256 had to be read from a different mirror, ret 0
FS_INFO AFTER IS 0x55555564cbc0
block 13577220816896 had to be read from a different mirror, ret 0
block 13577870934016 had to be read from a different mirror, ret 0
(...)
block 13577821011968 had to be read from a different mirror, ret 0
block 15645781196800 had to be read from a different mirror, ret 0
Checking root 2 bytenr 67387392
block 58720256 had to be read from a different mirror, ret 0
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
block 11970891186176 had to be read from a different mirror, ret 0
block 12511729680384 had to be read from a different mirror, ret 0
(...)
block 13577726541824 had to be read from a different mirror, ret 0
block 13577776611328 had to be read from a different mirror, ret 0
block 364637798400 had to be read from a different mirror, ret 0
block 364637962240 had to be read from a different mirror, ret 0
Checking root 7 bytenr 13577819963392
block 210337792 had to be read from a different mirror, ret 0
block 595607552 had to be read from a different mirror, ret 0
block 949698560 had to be read from a different mirror, ret 0
(...)
Checking root 165299 bytenr 13577191505920
block 364679708672 had to be read from a different mirror, ret 0
block 364713902080 had to be read from a different mirror, ret 0
(...)
block 13577821011968 had to be read from a different mirror, ret 0
block 15645781196800 had to be read from a different mirror, ret 0
Checking root 18446744073709551607 bytenr 13576823685120
Tree recovery finished, you can run check now
[Inferior 1 (process 5587) exited normally]
(gdb) quit


Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
block 15645878108160 had to be read from a different mirror, ret 0
Couldn't find the last root for 8
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
block 58720256 had to be read from a different mirror, ret 0
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
block 58720256 had to be read from a different mirror, ret 0
(...)
block 58720256 had to be read from a different mirror, ret 0
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
block 58720256 had to be read from a different mirror, ret 0
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
block 58720256 had to be read from a different mirror, ret 0
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks
checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d
block 11651824091136 had to be read from a different mirror, ret 0
checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188
block 606126080 had to be read from a different mirror, ret 0
checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858
block 15645807640576 had to be read from a different mirror, ret 0
checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82
block 364863324160 had to be read from a different mirror, ret 0
(...)
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
block 15645419667456 had to be read from a different mirror, ret 0
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
block 13577821011968 had to be read from a different mirror, ret 0
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
block 15645781196800 had to be read from a different mirror, ret 0
Clearing the extent root and re-init'ing the block groups
parent transid verify failed on 29442048 wanted 1619017 found 1619015
parent transid verify failed on 29442048 wanted 1619017 found 1619015
parent transid verify failed on 29442048 wanted 1619017 found 1619015
parent transid verify failed on 29556736 wanted 1619017 found 1619013
parent transid verify failed on 29556736 wanted 1619017 found 1619013
parent transid verify failed on 29556736 wanted 1619017 found 1619013
parent transid verify failed on 29589504 wanted 1619017 found 1619013
parent transid verify failed on 29589504 wanted 1619017 found 1619013
parent transid verify failed on 29589504 wanted 1619017 found 1619013
ERROR: Error adding block group
ERROR: commit_root already set when starting transaction
WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920
extent buffer leak: start 67469312 len 16384
extent buffer leak: start 29540352 len 16384
extent buffer leak: start 29540352 len 16384
WARNING: dirty eb leak (aborted trans): start 29540352 len 16384
extent buffer leak: start 29655040 len 16384
WARNING: dirty eb leak (aborted trans): start 29655040 len 16384
extent buffer leak: start 29589504 len 16384
extent buffer leak: start 29589504 len 16384
WARNING: dirty eb leak (aborted trans): start 29589504 len 16384
extent buffer leak: start 34504704 len 16384
WARNING: dirty eb leak (aborted trans): start 34504704 len 16384
Init extent tree failed
[Inferior 1 (process 11217) exited with code 0357]
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-24 22:38                                                                                                                                                                                 ` Marc MERLIN
@ 2022-04-24 22:56                                                                                                                                                                                   ` Josef Bacik
  2022-04-24 23:14                                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-24 22:56 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 6:38 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, Apr 24, 2022 at 05:26:11PM -0400, Josef Bacik wrote:
> > On Sun, Apr 24, 2022 at 5:20 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sun, Apr 24, 2022 at 05:13:22PM -0400, Josef Bacik wrote:
> > > > On Sun, Apr 24, 2022 at 5:07 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > >
> > > > > On Sun, Apr 24, 2022 at 05:01:38PM -0400, Josef Bacik wrote:
> > > > > > Wtf, it's reading the right bytenr, but it's not failing here but
> > > > > > failing when we do the init-extent-tree.  I've pushed something again
> > > > > > to force reads, maybe that's the problem, can you run tree-recover and
> > > > > > then init-extent-tree again?  Thanks,
> > > > >
> > > > > off-list:
> > > > > Seeing this, that kind of worried me so I stopped it. Is it ok to
> > > > > continue?
> > > >
> > > > Sorry, my printf was wrong, pushed.  Thanks,
> > >
> > > Better?
> > > Do I worry about those new "had to be read from a different mirror" ?
> > >
> >
> > Ooooh ok, the other mirror is bad, but it finds the right thing.  Ok
> > that makes sense.  I pushed another patch just to make sure.  Once
> > this is all done you'll need to run a scrub to fix the mirrors but
> > this is fine.  Thanks,
>
> No more crash, but:
> ERROR: Error adding block group
> ERROR: commit_root already set when starting transaction
>

I feel like this thing is purposefully changing itself between each
run so I can't get a grasp on wtf is going on.  I pushed some stuff,
lets see how that goes.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-24 22:56                                                                                                                                                                                   ` Josef Bacik
@ 2022-04-24 23:14                                                                                                                                                                                     ` Marc MERLIN
  2022-04-24 23:27                                                                                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-24 23:14 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 06:56:01PM -0400, Josef Bacik wrote:
> I feel like this thing is purposefully changing itself between each
> run so I can't get a grasp on wtf is going on.  I pushed some stuff,
> lets see how that goes.  Thanks,

After all the tests we did, is it possible that some damaged the FS
further?

Either way:
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue
init-extent-tree /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
block 15645878108160 had to be read from a different mirror, ret 0
Couldn't find the last root for 8
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
block 58720256 had to be read from a different mirror, ret 0
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
block 58720256 had to be read from a different mirror, ret 0
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
block 58720256 had to be read from a different mirror, ret 0
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
block 58720256 had to be read from a different mirror, ret 0
(...)
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks
block 11651824091136 had to be read from a different mirror, ret 0
block 606126080 had to be read from a different mirror, ret 0
block 15645807640576 had to be read from a different mirror, ret 0
block 364863324160 had to be read from a different mirror, ret 0
block 364970688512 had to be read from a different mirror, ret 0
block 15645178052608 had to be read from a different mirror, ret 0
(...)
inserting block group 352753549312
inserting block group 353827291136
Ignoring transid failure
Ignoring transid failure
ERROR: Error adding block group -17
ERROR: commit_root already set when starting transaction
WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920
extent buffer leak: start 67469312 len 16384
extent buffer leak: start 29540352 len 16384
WARNING: dirty eb leak (aborted trans): start 29540352 len 16384
extent buffer leak: start 29589504 len 16384
WARNING: dirty eb leak (aborted trans): start 29589504 len 16384
extent buffer leak: start 29655040 len 16384
WARNING: dirty eb leak (aborted trans): start 29655040 len 16384
Init extent tree failed
[Inferior 1 (process 1264) exited with code 0357]
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-24 23:14                                                                                                                                                                                     ` Marc MERLIN
@ 2022-04-24 23:27                                                                                                                                                                                       ` Josef Bacik
  2022-04-25  0:24                                                                                                                                                                                         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-24 23:27 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 7:14 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, Apr 24, 2022 at 06:56:01PM -0400, Josef Bacik wrote:
> > I feel like this thing is purposefully changing itself between each
> > run so I can't get a grasp on wtf is going on.  I pushed some stuff,
> > lets see how that goes.  Thanks,
>
> After all the tests we did, is it possible that some damaged the FS
> further?
>

That's the thing, we're literally deleting the entire tree and
starting over, it should do the same thing every time.  I pushed
another fix, I think I've been messing up the buffers and that's why
we're getting random values. Lets try this again,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-24 23:27                                                                                                                                                                                       ` Josef Bacik
@ 2022-04-25  0:24                                                                                                                                                                                         ` Marc MERLIN
  2022-04-25  0:36                                                                                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-25  0:24 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 07:27:02PM -0400, Josef Bacik wrote:
> On Sun, Apr 24, 2022 at 7:14 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sun, Apr 24, 2022 at 06:56:01PM -0400, Josef Bacik wrote:
> > > I feel like this thing is purposefully changing itself between each
> > > run so I can't get a grasp on wtf is going on.  I pushed some stuff,
> > > lets see how that goes.  Thanks,
> >
> > After all the tests we did, is it possible that some damaged the FS
> > further?
> >
> 
> That's the thing, we're literally deleting the entire tree and
> starting over, it should do the same thing every time.  I pushed
> another fix, I think I've been messing up the buffers and that's why
> we're getting random values. Lets try this again,

Gotcha.

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
Couldn't find the last root for 8
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
(...)
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups
inserting block group 12582912
inserting block group 20971520
inserting block group 29360128
inserting block group 1103101952
(...)
inserting block group 345237356544
inserting block group 346311098368
inserting block group 347384840192
inserting block group 348458582016
inserting block group 349532323840
inserting block group 350606065664
inserting block group 351679807488
inserting block group 352753549312
inserting block group 353827291136
Ignoring transid failure
Ignoring transid failure
ERROR: Error adding block group -17
ERROR: commit_root already set when starting transaction
WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920
extent buffer leak: start 67469312 len 16384
extent buffer leak: start 29540352 len 16384
WARNING: dirty eb leak (aborted trans): start 29540352 len 16384
extent buffer leak: start 29589504 len 16384
WARNING: dirty eb leak (aborted trans): start 29589504 len 16384
extent buffer leak: start 29655040 len 16384
WARNING: dirty eb leak (aborted trans): start 29655040 len 16384
Init extent tree failed
[Inferior 1 (process 6259) exited with code 0357]
(gdb) 


I ran it a second time and got the same output

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-25  0:24                                                                                                                                                                                         ` Marc MERLIN
@ 2022-04-25  0:36                                                                                                                                                                                           ` Josef Bacik
  2022-04-26  0:28                                                                                                                                                                                             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-25  0:36 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 8:24 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, Apr 24, 2022 at 07:27:02PM -0400, Josef Bacik wrote:
> > On Sun, Apr 24, 2022 at 7:14 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sun, Apr 24, 2022 at 06:56:01PM -0400, Josef Bacik wrote:
> > > > I feel like this thing is purposefully changing itself between each
> > > > run so I can't get a grasp on wtf is going on.  I pushed some stuff,
> > > > lets see how that goes.  Thanks,
> > >
> > > After all the tests we did, is it possible that some damaged the FS
> > > further?
> > >
> >
> > That's the thing, we're literally deleting the entire tree and
> > starting over, it should do the same thing every time.  I pushed
> > another fix, I think I've been messing up the buffers and that's why
> > we're getting random values. Lets try this again,
>
> Gotcha.
>
> (gdb) run rescue init-extent-tree /dev/mapper/dshelf1
> Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> FS_INFO IS 0x55555564cbc0
> JOSEF: root 9
> checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
> Couldn't find the last root for 8
> checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
> checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
> checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
> checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
> checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
> checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
> (...)
> checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
> FS_INFO AFTER IS 0x55555564cbc0
> Walking all our trees and pinning down the currently accessible blocks
> Clearing the extent root and re-init'ing the block groups
> inserting block group 12582912
> inserting block group 20971520
> inserting block group 29360128
> inserting block group 1103101952
> (...)
> inserting block group 345237356544
> inserting block group 346311098368
> inserting block group 347384840192
> inserting block group 348458582016
> inserting block group 349532323840
> inserting block group 350606065664
> inserting block group 351679807488
> inserting block group 352753549312
> inserting block group 353827291136
> Ignoring transid failure
> Ignoring transid failure
> ERROR: Error adding block group -17
> ERROR: commit_root already set when starting transaction
> WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920
> extent buffer leak: start 67469312 len 16384
> extent buffer leak: start 29540352 len 16384
> WARNING: dirty eb leak (aborted trans): start 29540352 len 16384
> extent buffer leak: start 29589504 len 16384
> WARNING: dirty eb leak (aborted trans): start 29589504 len 16384
> extent buffer leak: start 29655040 len 16384
> WARNING: dirty eb leak (aborted trans): start 29655040 len 16384
> Init extent tree failed
> [Inferior 1 (process 6259) exited with code 0357]
> (gdb)
>
>
> I ran it a second time and got the same output

Well thats something at least, I've rigged it up to dump the leaf so I
can see wtf is going on here.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-25  0:36                                                                                                                                                                                           ` Josef Bacik
@ 2022-04-26  0:28                                                                                                                                                                                             ` Marc MERLIN
  2022-04-26 20:43                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-26  0:28 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Apr 24, 2022 at 08:36:42PM -0400, Josef Bacik wrote:
> > I ran it a second time and got the same output
> 
> Well thats something at least, I've rigged it up to dump the leaf so I
> can see wtf is going on here.  Thanks,

Sorry for the delay, took a while to run:
(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

FS_INFO IS 0x55555564cbc0
JOSEF: root 9
Couldn't find the last root for 8
inserting block group 253969301504
        item 33 key (328057487360 BLOCK_GROUP_ITEM 1073741824) itemoff 15467 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 173 key (477844471808 BLOCK_GROUP_ITEM 1073741824) itemoff 12107 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 174 key (478918213632 BLOCK_GROUP_ITEM 1073741824) itemoff 12083 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 175 key (479991955456 BLOCK_GROUP_ITEM 1073741824) itemoff 12059 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 176 key (481065697280 BLOCK_GROUP_ITEM 1073741824) itemoff 12035 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 177 key (482139439104 BLOCK_GROUP_ITEM 1073741824) itemoff 12011 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 178 key (483213180928 BLOCK_GROUP_ITEM 1073741824) itemoff 11987 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 179 key (484286922752 BLOCK_GROUP_ITEM 1073741824) itemoff 11963 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 180 key (485360664576 BLOCK_GROUP_ITEM 1073741824) itemoff 11939 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 181 key (486434406400 BLOCK_GROUP_ITEM 1073741824) itemoff 11915 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 182 key (487508148224 BLOCK_GROUP_ITEM 1073741824) itemoff 11891 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 183 key (488581890048 BLOCK_GROUP_ITEM 1073741824) itemoff 11867 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 184 key (489655631872 BLOCK_GROUP_ITEM 1073741824) itemoff 11843 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 185 key (490729373696 BLOCK_GROUP_ITEM 1073741824) itemoff 11819 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 186 key (491803115520 BLOCK_GROUP_ITEM 1073741824) itemoff 11795 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 187 key (492876857344 BLOCK_GROUP_ITEM 1073741824) itemoff 11771 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 188 key (493950599168 BLOCK_GROUP_ITEM 1073741824) itemoff 11747 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 189 key (495024340992 BLOCK_GROUP_ITEM 1073741824) itemoff 11723 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 190 key (496098082816 BLOCK_GROUP_ITEM 1073741824) itemoff 11699 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 191 key (497171824640 BLOCK_GROUP_ITEM 1073741824) itemoff 11675 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 192 key (498245566464 BLOCK_GROUP_ITEM 1073741824) itemoff 11651 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 193 key (499319308288 BLOCK_GROUP_ITEM 1073741824) itemoff 11627 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 194 key (500393050112 BLOCK_GROUP_ITEM 1073741824) itemoff 11603 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 195 key (501466791936 BLOCK_GROUP_ITEM 1073741824) itemoff 11579 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 196 key (502540533760 BLOCK_GROUP_ITEM 1073741824) itemoff 11555 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 197 key (503614275584 BLOCK_GROUP_ITEM 1073741824) itemoff 11531 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 198 key (504688017408 BLOCK_GROUP_ITEM 1073741824) itemoff 11507 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 199 key (505761759232 BLOCK_GROUP_ITEM 1073741824) itemoff 11483 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 200 key (506835501056 BLOCK_GROUP_ITEM 1073741824) itemoff 11459 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 201 key (507909242880 BLOCK_GROUP_ITEM 1073741824) itemoff 11435 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 202 key (508982984704 BLOCK_GROUP_ITEM 1073741824) itemoff 11411 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 203 key (510056726528 BLOCK_GROUP_ITEM 1073741824) itemoff 11387 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 204 key (511130468352 BLOCK_GROUP_ITEM 1073741824) itemoff 11363 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 205 key (512204210176 BLOCK_GROUP_ITEM 1073741824) itemoff 11339 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 206 key (513277952000 BLOCK_GROUP_ITEM 1073741824) itemoff 11315 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 207 key (514351693824 BLOCK_GROUP_ITEM 1073741824) itemoff 11291 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 208 key (515425435648 BLOCK_GROUP_ITEM 1073741824) itemoff 11267 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 209 key (516499177472 BLOCK_GROUP_ITEM 1073741824) itemoff 11243 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 210 key (517572919296 BLOCK_GROUP_ITEM 1073741824) itemoff 11219 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 211 key (518646661120 BLOCK_GROUP_ITEM 1073741824) itemoff 11195 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 212 key (519720402944 BLOCK_GROUP_ITEM 1073741824) itemoff 11171 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 213 key (520794144768 BLOCK_GROUP_ITEM 1073741824) itemoff 11147 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 214 key (521867886592 BLOCK_GROUP_ITEM 1073741824) itemoff 11123 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 215 key (522941628416 BLOCK_GROUP_ITEM 1073741824) itemoff 11099 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 216 key (524015370240 BLOCK_GROUP_ITEM 1073741824) itemoff 11075 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 217 key (525089112064 BLOCK_GROUP_ITEM 1073741824) itemoff 11051 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
ERROR: Error adding block group -17
ERROR: commit_root already set when starting transaction
WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920
extent buffer leak: start 67469312 len 16384
extent buffer leak: start 29540352 len 16384
WARNING: dirty eb leak (aborted trans): start 29540352 len 16384
extent buffer leak: start 29589504 len 16384
WARNING: dirty eb leak (aborted trans): start 29589504 len 16384
extent buffer leak: start 29655040 len 16384
WARNING: dirty eb leak (aborted trans): start 29655040 len 16384
Init extent tree failed
[Inferior 1 (process 24213) exited with code 0357]

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-26  0:28                                                                                                                                                                                             ` Marc MERLIN
@ 2022-04-26 20:43                                                                                                                                                                                               ` Marc MERLIN
  2022-04-26 21:20                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-26 20:43 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

Generally would you say we're still on the right path and helping your
recovery tools getting better, or is it getting close or to the time
where I should restore from backups?

On Mon, Apr 25, 2022 at 05:28:04PM -0700, Marc MERLIN wrote:
> On Sun, Apr 24, 2022 at 08:36:42PM -0400, Josef Bacik wrote:
> > > I ran it a second time and got the same output
> > 
> > Well thats something at least, I've rigged it up to dump the leaf so I
> > can see wtf is going on here.  Thanks,
> 
> Sorry for the delay, took a while to run:
> (gdb) run rescue init-extent-tree /dev/mapper/dshelf1
> Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> 
> FS_INFO IS 0x55555564cbc0
> JOSEF: root 9
> Couldn't find the last root for 8
> inserting block group 253969301504
>         item 33 key (328057487360 BLOCK_GROUP_ITEM 1073741824) itemoff 15467 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 173 key (477844471808 BLOCK_GROUP_ITEM 1073741824) itemoff 12107 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 174 key (478918213632 BLOCK_GROUP_ITEM 1073741824) itemoff 12083 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 175 key (479991955456 BLOCK_GROUP_ITEM 1073741824) itemoff 12059 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 176 key (481065697280 BLOCK_GROUP_ITEM 1073741824) itemoff 12035 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 177 key (482139439104 BLOCK_GROUP_ITEM 1073741824) itemoff 12011 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 178 key (483213180928 BLOCK_GROUP_ITEM 1073741824) itemoff 11987 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 179 key (484286922752 BLOCK_GROUP_ITEM 1073741824) itemoff 11963 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 180 key (485360664576 BLOCK_GROUP_ITEM 1073741824) itemoff 11939 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 181 key (486434406400 BLOCK_GROUP_ITEM 1073741824) itemoff 11915 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 182 key (487508148224 BLOCK_GROUP_ITEM 1073741824) itemoff 11891 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 183 key (488581890048 BLOCK_GROUP_ITEM 1073741824) itemoff 11867 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 184 key (489655631872 BLOCK_GROUP_ITEM 1073741824) itemoff 11843 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 185 key (490729373696 BLOCK_GROUP_ITEM 1073741824) itemoff 11819 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 186 key (491803115520 BLOCK_GROUP_ITEM 1073741824) itemoff 11795 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 187 key (492876857344 BLOCK_GROUP_ITEM 1073741824) itemoff 11771 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 188 key (493950599168 BLOCK_GROUP_ITEM 1073741824) itemoff 11747 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 189 key (495024340992 BLOCK_GROUP_ITEM 1073741824) itemoff 11723 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 190 key (496098082816 BLOCK_GROUP_ITEM 1073741824) itemoff 11699 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 191 key (497171824640 BLOCK_GROUP_ITEM 1073741824) itemoff 11675 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 192 key (498245566464 BLOCK_GROUP_ITEM 1073741824) itemoff 11651 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 193 key (499319308288 BLOCK_GROUP_ITEM 1073741824) itemoff 11627 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 194 key (500393050112 BLOCK_GROUP_ITEM 1073741824) itemoff 11603 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 195 key (501466791936 BLOCK_GROUP_ITEM 1073741824) itemoff 11579 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 196 key (502540533760 BLOCK_GROUP_ITEM 1073741824) itemoff 11555 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 197 key (503614275584 BLOCK_GROUP_ITEM 1073741824) itemoff 11531 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 198 key (504688017408 BLOCK_GROUP_ITEM 1073741824) itemoff 11507 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 199 key (505761759232 BLOCK_GROUP_ITEM 1073741824) itemoff 11483 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 200 key (506835501056 BLOCK_GROUP_ITEM 1073741824) itemoff 11459 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 201 key (507909242880 BLOCK_GROUP_ITEM 1073741824) itemoff 11435 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 202 key (508982984704 BLOCK_GROUP_ITEM 1073741824) itemoff 11411 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 203 key (510056726528 BLOCK_GROUP_ITEM 1073741824) itemoff 11387 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 204 key (511130468352 BLOCK_GROUP_ITEM 1073741824) itemoff 11363 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 205 key (512204210176 BLOCK_GROUP_ITEM 1073741824) itemoff 11339 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 206 key (513277952000 BLOCK_GROUP_ITEM 1073741824) itemoff 11315 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 207 key (514351693824 BLOCK_GROUP_ITEM 1073741824) itemoff 11291 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 208 key (515425435648 BLOCK_GROUP_ITEM 1073741824) itemoff 11267 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 209 key (516499177472 BLOCK_GROUP_ITEM 1073741824) itemoff 11243 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 210 key (517572919296 BLOCK_GROUP_ITEM 1073741824) itemoff 11219 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 211 key (518646661120 BLOCK_GROUP_ITEM 1073741824) itemoff 11195 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 212 key (519720402944 BLOCK_GROUP_ITEM 1073741824) itemoff 11171 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 213 key (520794144768 BLOCK_GROUP_ITEM 1073741824) itemoff 11147 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 214 key (521867886592 BLOCK_GROUP_ITEM 1073741824) itemoff 11123 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 215 key (522941628416 BLOCK_GROUP_ITEM 1073741824) itemoff 11099 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 216 key (524015370240 BLOCK_GROUP_ITEM 1073741824) itemoff 11075 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
>         item 217 key (525089112064 BLOCK_GROUP_ITEM 1073741824) itemoff 11051 itemsize 24
>                 block group used 0 chunk_objectid 256 flags DATA|single
> ERROR: Error adding block group -17
> ERROR: commit_root already set when starting transaction
> WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920
> extent buffer leak: start 67469312 len 16384
> extent buffer leak: start 29540352 len 16384
> WARNING: dirty eb leak (aborted trans): start 29540352 len 16384
> extent buffer leak: start 29589504 len 16384
> WARNING: dirty eb leak (aborted trans): start 29589504 len 16384
> extent buffer leak: start 29655040 len 16384
> WARNING: dirty eb leak (aborted trans): start 29655040 len 16384
> Init extent tree failed
> [Inferior 1 (process 24213) exited with code 0357]
> 
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>  
> Home page: http://marc.merlins.org/  
> 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-26 20:43                                                                                                                                                                                               ` Marc MERLIN
@ 2022-04-26 21:20                                                                                                                                                                                                 ` Josef Bacik
  2022-04-26 21:36                                                                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-26 21:20 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 26, 2022 at 4:43 PM Marc MERLIN <marc@merlins.org> wrote:
>
> Generally would you say we're still on the right path and helping your
> recovery tools getting better, or is it getting close or to the time
> where I should restore from backups?
>

Yup sorry for the radio silence, loads of meetings today, but good
news is I've reproduced your problem locally, so I'm trying to hammer
it out.  I hope to have something useful for you today.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-26 21:20                                                                                                                                                                                                 ` Josef Bacik
@ 2022-04-26 21:36                                                                                                                                                                                                   ` Josef Bacik
  2022-04-27  3:54                                                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-26 21:36 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 26, 2022 at 5:20 PM Josef Bacik <josef@toxicpanda.com> wrote:
>
> On Tue, Apr 26, 2022 at 4:43 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > Generally would you say we're still on the right path and helping your
> > recovery tools getting better, or is it getting close or to the time
> > where I should restore from backups?
> >
>
> Yup sorry for the radio silence, loads of meetings today, but good
> news is I've reproduced your problem locally, so I'm trying to hammer
> it out.  I hope to have something useful for you today.  Thanks,
>

Sigh I'm dumb as fuck, can you pull and re-run tree-recover just to
make sure any stupidity I've caused is undone, and then run rescue
init-extent-tree and then we can go from there?  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-26 21:36                                                                                                                                                                                                   ` Josef Bacik
@ 2022-04-27  3:54                                                                                                                                                                                                     ` Marc MERLIN
  2022-04-27 14:44                                                                                                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-27  3:54 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Apr 26, 2022 at 05:36:28PM -0400, Josef Bacik wrote:
> On Tue, Apr 26, 2022 at 5:20 PM Josef Bacik <josef@toxicpanda.com> wrote:
> >
> > On Tue, Apr 26, 2022 at 4:43 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > Generally would you say we're still on the right path and helping your
> > > recovery tools getting better, or is it getting close or to the time
> > > where I should restore from backups?
> > >
> >
> > Yup sorry for the radio silence, loads of meetings today, but good
> > news is I've reproduced your problem locally, so I'm trying to hammer
> > it out.  I hope to have something useful for you today.  Thanks,
> 
> Sigh I'm dumb as fuck, can you pull and re-run tree-recover just to
> make sure any stupidity I've caused is undone, and then run rescue
> init-extent-tree and then we can go from there?  Thanks,

FS_INFO IS 0x55555564cbc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55555564cbc0
Checking root 2 bytenr 67387392
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 13577819963392
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736
Checking root 11222 bytenr 15645261905920
Checking root 11223 bytenr 13576862547968
Checking root 11224 bytenr 13577126182912
Checking root 159785 bytenr 6781490577408
Checking root 159787 bytenr 15645908385792
Checking root 160494 bytenr 6781491265536
Checking root 160496 bytenr 11822309965824
Checking root 161197 bytenr 6781492101120
Checking root 161199 bytenr 13576850833408
Checking root 162628 bytenr 15645764812800
Checking root 162632 bytenr 6781492756480
Checking root 162645 bytenr 5809981095936
Checking root 163298 bytenr 15645124263936
Checking root 163302 bytenr 6781495197696
Checking root 163303 bytenr 15645365993472
Checking root 163316 bytenr 6781496393728
Checking root 163318 bytenr 15645980491776
Checking root 163916 bytenr 11822437826560
Checking root 163920 bytenr 11971021275136
Checking root 163921 bytenr 11971073802240
Checking root 164620 bytenr 15645434036224
Checking root 164624 bytenr 15645502210048
Checking root 164633 bytenr 15645526884352
Checking root 165098 bytenr 11970667446272
Checking root 165100 bytenr 11970733621248
Checking root 165198 bytenr 12511656394752
Checking root 165200 bytenr 12511677972480
Checking root 165294 bytenr 13576901328896
Checking root 165298 bytenr 13577133326336
Checking root 165299 bytenr 13577191505920
Checking root 18446744073709551607 bytenr 13576823685120
Tree recovery finished, you can run check now
[Inferior 1 (process 28238) exited normally]


(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55555564cbc0
Checking root 2 bytenr 67387392
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 13577819963392
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736
Checking root 11222 bytenr 15645261905920
Checking root 11223 bytenr 13576862547968
Checking root 11224 bytenr 13577126182912
Checking root 159785 bytenr 6781490577408
Checking root 159787 bytenr 15645908385792
Checking root 160494 bytenr 6781491265536
Checking root 160496 bytenr 11822309965824
Checking root 161197 bytenr 6781492101120
Checking root 161199 bytenr 13576850833408
Checking root 162628 bytenr 15645764812800
Checking root 162632 bytenr 6781492756480
Checking root 162645 bytenr 5809981095936
Checking root 163298 bytenr 15645124263936
Checking root 163302 bytenr 6781495197696
Checking root 163303 bytenr 15645365993472
Checking root 163316 bytenr 6781496393728
Checking root 163318 bytenr 15645980491776
Checking root 163916 bytenr 11822437826560
Checking root 163920 bytenr 11971021275136
Checking root 163921 bytenr 11971073802240
Checking root 164620 bytenr 15645434036224
Checking root 164624 bytenr 15645502210048
Checking root 164633 bytenr 15645526884352
Checking root 165098 bytenr 11970667446272
Checking root 165100 bytenr 11970733621248
Checking root 165198 bytenr 12511656394752
Checking root 165200 bytenr 12511677972480
Checking root 165294 bytenr 13576901328896
Checking root 165298 bytenr 13577133326336
Checking root 165299 bytenr 13577191505920
Checking root 18446744073709551607 bytenr 13576823685120
Tree recovery finished, you can run check now
[Inferior 1 (process 28238) exited normally]
(gdb) 
        item 210 key (517572919296 BLOCK_GROUP_ITEM 1073741824) itemoff 11219 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 211 key (518646661120 BLOCK_GROUP_ITEM 1073741824) itemoff 11195 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 212 key (519720402944 BLOCK_GROUP_ITEM 1073741824) itemoff 11171 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 214 key (521867886592 BLOCK_GROUP_ITEM 1073741824) itemoff 11123 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 215 key (522941628416 BLOCK_GROUP_ITEM 1073741824) itemoff 11099 itemsize 24
inserting block group 15731991248896
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1228800 of 0 possible bytesFailed to find [7750833868800, 168, 262144]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffdd00, comp=comp@entry=0x55555559ae9e <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffdd18) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffdd00, comp=comp@entry=0x55555559ae9e <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffdd18) at common/rbtree-utils.c:48
#1  0x000055555559b0a2 in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=99713024)
    at common/extent-cache.c:179
#2  0x0000555555584c30 in set_extent_bits (tree=0x10000000060, start=99713024, end=99713023, bits=bits@entry=1)
    at kernel-shared/extent_io.c:380
#3  0x0000555555584e63 in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x0000555555585846 in set_extent_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/extent_io.c:976
#5  0x000055555557bc1e in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee3d in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434246, parent=0, 
    iref=0xffffffffffffffe4, path=0x555555b92bd0, root=<optimized out>) at kernel-shared/extent-tree.c:1084
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434246, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b92bd0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1196
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555558a89620, root=root@entry=0x55555564cde0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0)
    at kernel-shared/extent-tree.c:1261
#9  0x00005555555dfb81 in process_eb (trans=trans@entry=0x555558a89620, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559cfce80, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:502
#10 0x00005555555dfc90 in process_eb (trans=trans@entry=0x555558a89620, root=root@entry=0x55555564cde0, eb=0x555555944100, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:577
#11 0x00005555555dfef8 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:648
#12 0x00005555555e0202 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:784
#13 0x00005555555d7a0c in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffe3c0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#17 main (argc=3, argv=0x7fffffffe3c0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-27  3:54                                                                                                                                                                                                     ` Marc MERLIN
@ 2022-04-27 14:44                                                                                                                                                                                                       ` Josef Bacik
  2022-04-27 16:34                                                                                                                                                                                                         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-27 14:44 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Apr 26, 2022 at 11:54 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Apr 26, 2022 at 05:36:28PM -0400, Josef Bacik wrote:
> > On Tue, Apr 26, 2022 at 5:20 PM Josef Bacik <josef@toxicpanda.com> wrote:
> > >
> > > On Tue, Apr 26, 2022 at 4:43 PM Marc MERLIN <marc@merlins.org> wrote:
> > > >
> > > > Generally would you say we're still on the right path and helping your
> > > > recovery tools getting better, or is it getting close or to the time
> > > > where I should restore from backups?
> > > >
> > >
> > > Yup sorry for the radio silence, loads of meetings today, but good
> > > news is I've reproduced your problem locally, so I'm trying to hammer
> > > it out.  I hope to have something useful for you today.  Thanks,
> >
> > Sigh I'm dumb as fuck, can you pull and re-run tree-recover just to
> > make sure any stupidity I've caused is undone, and then run rescue
> > init-extent-tree and then we can go from there?  Thanks,
>

Ok back to this problem again, I've added some debugging.
Unfortunately the real bytenr is getting lost, so the new debugging
will print the actual bytenr we're trying to add, and then I can add
more targeted debugging to figure out whats going on.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-27 14:44                                                                                                                                                                                                       ` Josef Bacik
@ 2022-04-27 16:34                                                                                                                                                                                                         ` Marc MERLIN
  2022-04-27 17:49                                                                                                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-27 16:34 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 10:44:26AM -0400, Josef Bacik wrote:
> On Tue, Apr 26, 2022 at 11:54 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Tue, Apr 26, 2022 at 05:36:28PM -0400, Josef Bacik wrote:
> > > On Tue, Apr 26, 2022 at 5:20 PM Josef Bacik <josef@toxicpanda.com> wrote:
> > > >
> > > > On Tue, Apr 26, 2022 at 4:43 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > >
> > > > > Generally would you say we're still on the right path and helping your
> > > > > recovery tools getting better, or is it getting close or to the time
> > > > > where I should restore from backups?
> > > > >
> > > >
> > > > Yup sorry for the radio silence, loads of meetings today, but good
> > > > news is I've reproduced your problem locally, so I'm trying to hammer
> > > > it out.  I hope to have something useful for you today.  Thanks,
> > >
> > > Sigh I'm dumb as fuck, can you pull and re-run tree-recover just to
> > > make sure any stupidity I've caused is undone, and then run rescue
> > > init-extent-tree and then we can go from there?  Thanks,
> 
> Ok back to this problem again, I've added some debugging.
> Unfortunately the real bytenr is getting lost, so the new debugging
> will print the actual bytenr we're trying to add, and then I can add
> more targeted debugging to figure out whats going on.  Thanks,

Small typo:
  409 |  prinf("doing insert of %llu\n", key->objectid);
        |  ^~~~~
	|  printf

Result:
doing insert of 12233379401728
Failed to find [7750833868800, 168, 262144]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffdd00, comp=comp@entry=0x55555559aea8 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffdd18) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffdd00, comp=comp@entry=0x55555559aea8 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffdd18) at common/rbtree-utils.c:48
#1  0x000055555559b0ac in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=100564992)
    at common/extent-cache.c:179
#2  0x0000555555584c3a in set_extent_bits (tree=0x10000000060, start=100564992, end=100564991, bits=bits@entry=1)
    at kernel-shared/extent_io.c:380
#3  0x0000555555584e6d in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x0000555555585850 in set_extent_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/extent_io.c:976
#5  0x000055555557bc1e in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee47 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434256, parent=0, 
    iref=0xffffffffffffffe4, path=0x555555b8cda0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434256, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b8cda0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555558a8d760, root=root@entry=0x55555564cde0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555dfba4 in process_eb (trans=trans@entry=0x555558a8d760, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d00cc0, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:503
#10 0x00005555555dfcb3 in process_eb (trans=trans@entry=0x555558a8d760, root=root@entry=0x55555564cde0, eb=0x555555944100, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:578
#11 0x00005555555dff1b in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:649
#12 0x00005555555e0225 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:785
#13 0x00005555555d7a16 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffe

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-27 16:34                                                                                                                                                                                                         ` Marc MERLIN
@ 2022-04-27 17:49                                                                                                                                                                                                           ` Josef Bacik
  2022-04-27 18:24                                                                                                                                                                                                             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-27 17:49 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 12:34 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Apr 27, 2022 at 10:44:26AM -0400, Josef Bacik wrote:
> > On Tue, Apr 26, 2022 at 11:54 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Tue, Apr 26, 2022 at 05:36:28PM -0400, Josef Bacik wrote:
> > > > On Tue, Apr 26, 2022 at 5:20 PM Josef Bacik <josef@toxicpanda.com> wrote:
> > > > >
> > > > > On Tue, Apr 26, 2022 at 4:43 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > > >
> > > > > > Generally would you say we're still on the right path and helping your
> > > > > > recovery tools getting better, or is it getting close or to the time
> > > > > > where I should restore from backups?
> > > > > >
> > > > >
> > > > > Yup sorry for the radio silence, loads of meetings today, but good
> > > > > news is I've reproduced your problem locally, so I'm trying to hammer
> > > > > it out.  I hope to have something useful for you today.  Thanks,
> > > >
> > > > Sigh I'm dumb as fuck, can you pull and re-run tree-recover just to
> > > > make sure any stupidity I've caused is undone, and then run rescue
> > > > init-extent-tree and then we can go from there?  Thanks,
> >
> > Ok back to this problem again, I've added some debugging.
> > Unfortunately the real bytenr is getting lost, so the new debugging
> > will print the actual bytenr we're trying to add, and then I can add
> > more targeted debugging to figure out whats going on.  Thanks,
>
> Small typo:
>   409 |  prinf("doing insert of %llu\n", key->objectid);
>         |  ^~~~~
>         |  printf
>
> Result:
> doing insert of 12233379401728
> Failed to find [7750833868800, 168, 262144]

Oh it's data, interesting.  Pushed some more debugging.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-27 17:49                                                                                                                                                                                                           ` Josef Bacik
@ 2022-04-27 18:24                                                                                                                                                                                                             ` Marc MERLIN
  2022-04-27 20:21                                                                                                                                                                                                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-27 18:24 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 01:49:46PM -0400, Josef Bacik wrote:
> > Result:
> > doing insert of 12233379401728
> > Failed to find [7750833868800, 168, 262144]
> 
> Oh it's data, interesting.  Pushed some more debugging.  Thanks,

inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1130496 of 0 possible bytesdoing an insert that overlaps our bytenr 7750833627136 262144
processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr??
Failed to find [7750833868800, 168, 262144]
processed 1245184 of 0 possible bytesIgnoring transid failure
kernel-shared/ctree.c:1940: leaf_space_used: Warning: assertion `data_len < 0` failed, value 1
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x1d13f)[0x55555557113f]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_leaf_free_space+0xd3)[0x555555573b36]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x1fbf3)[0x555555573bf3]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x2083b)[0x55555557483b]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_search_slot+0x11cd)[0x5555555767c9]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_insert_empty_items+0x76)[0x555555576e6b]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8b97d)[0x5555555df97d]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bc04)[0x5555555dfc04]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bcfb)[0x5555555dfcfb]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bf59)[0x5555555dff59]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_init_extent_tree+0x22d)[0x5555555e0263]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x83a16)[0x5555555d7a16]
/var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b]
/var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd]
/var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]
kernel-shared/ctree.c:1940: leaf_space_used: Warning: assertion `data_len < 0` failed, value 1
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x1d13f)[0x55555557113f]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_leaf_free_space+0xd3)[0x555555573b36]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x1fc38)[0x555555573c38]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x2083b)[0x55555557483b]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_search_slot+0x11cd)[0x5555555767c9]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_insert_empty_items+0x76)[0x555555576e6b]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8b97d)[0x5555555df97d]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bc04)[0x5555555dfc04]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bcfb)[0x5555555dfcfb]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bf59)[0x5555555dff59]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_init_extent_tree+0x22d)[0x5555555e0263]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x83a16)[0x5555555d7a16]
/var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b]
/var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd]
/var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x1000000000060, key=key@entry=0x7fffffffd8f0, comp=comp@entry=0x55555559aea8 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd908) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x1000000000060, key=key@entry=0x7fffffffd8f0, 
    comp=comp@entry=0x55555559aea8 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd908) at common/rbtree-utils.c:48
#1  0x000055555559b0ac in search_cache_extent (tree=tree@entry=0x1000000000060, start=start@entry=1) at common/extent-cache.c:179
#2  0x0000555555584c3a in set_extent_bits (tree=0x1000000000060, start=1, end=0, bits=bits@entry=1) at kernel-shared/extent_io.c:380
#3  0x0000555555584e6d in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x0000555555585850 in set_extent_buffer_dirty (eb=0x555555708580) at kernel-shared/extent_io.c:976
#5  0x000055555557bc1e in btrfs_mark_buffer_dirty (eb=<optimized out>) at kernel-shared/disk-io.c:2224
#6  0x00005555555740c3 in push_leaf_right (trans=trans@entry=0x555558a8d700, root=root@entry=0x5555559547d0, 
    path=path@entry=0x7fffffffde40, data_size=data_size@entry=49, empty=empty@entry=0) at kernel-shared/ctree.c:2106
#7  0x000055555557483b in split_leaf (trans=trans@entry=0x555558a8d700, root=root@entry=0x5555559547d0, 
    ins_key=ins_key@entry=0x7fffffffdf60, path=path@entry=0x7fffffffde40, data_size=data_size@entry=49, extend=extend@entry=0)
    at kernel-shared/ctree.c:2385
#8  0x00005555555767c9 in btrfs_search_slot (trans=0x555558a8d700, root=root@entry=0x5555559547d0, key=key@entry=0x7fffffffdf60, 
    p=p@entry=0x7fffffffde40, ins_len=ins_len@entry=49, cow=cow@entry=1) at kernel-shared/ctree.c:1432
#9  0x0000555555576e6b in btrfs_insert_empty_items (trans=trans@entry=0x555558a8d700, root=root@entry=0x5555559547d0, 
    path=path@entry=0x7fffffffde40, cpu_key=cpu_key@entry=0x7fffffffdf60, data_size=data_size@entry=0x7fffffffde3c, nr=nr@entry=1)
    at kernel-shared/ctree.c:2820
#10 0x00005555555df97d in btrfs_insert_empty_item (data_size=<optimized out>, key=0x7fffffffdf60, path=0x7fffffffde40, 
    root=0x5555559547d0, trans=0x555558a8d700) at ./kernel-shared/ctree.h:2780
#11 insert_empty_extent (trans=trans@entry=0x555558a8d700, key=key@entry=0x7fffffffdf60, generation=generation@entry=1590260, 
    flags=flags@entry=1) at cmds/rescue-init-extent-tree.c:435
#12 0x00005555555dfc04 in process_eb (trans=trans@entry=0x555558a8d700, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d04fe0, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:508
#13 0x00005555555dfcfb in process_eb (trans=trans@entry=0x555558a8d700, root=root@entry=0x55555564cde0, eb=0x555555944100, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:594
#14 0x00005555555dff59 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:665
#15 0x00005555555e0263 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:801
#16 0x00005555555d7a16 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#17 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#18 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152
#19 0x000055555556c275 in cmd_execute (argv=0x7fffffffe3c0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#20 main (argc=3, argv=0x7fffffffe3c0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-27 18:24                                                                                                                                                                                                             ` Marc MERLIN
@ 2022-04-27 20:21                                                                                                                                                                                                               ` Josef Bacik
  2022-04-27 21:02                                                                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-27 20:21 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 2:24 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Apr 27, 2022 at 01:49:46PM -0400, Josef Bacik wrote:
> > > Result:
> > > doing insert of 12233379401728
> > > Failed to find [7750833868800, 168, 262144]
> >
> > Oh it's data, interesting.  Pushed some more debugging.  Thanks,
>
> inserting block group 15840439173120
> inserting block group 15842586656768
> processed 1556480 of 0 possible bytes
> processed 1130496 of 0 possible bytesdoing an insert that overlaps our bytenr 7750833627136 262144
> processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr??

Well crap, you have bytenrs that overlap.  I've adjusted
init-extent-tree to dump the paths of the files that have overlapping
extents, as well as the keys.  Run it again so we know what files are
fucked. From there you need to tell me which one you don't care about
and are willing to delete, and then I'll give you the command you need
to remove that bad extent and then we can go again.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-27 20:21                                                                                                                                                                                                               ` Josef Bacik
@ 2022-04-27 21:02                                                                                                                                                                                                                 ` Marc MERLIN
  2022-04-27 21:11                                                                                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-27 21:02 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 04:21:31PM -0400, Josef Bacik wrote:
> On Wed, Apr 27, 2022 at 2:24 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Wed, Apr 27, 2022 at 01:49:46PM -0400, Josef Bacik wrote:
> > > > Result:
> > > > doing insert of 12233379401728
> > > > Failed to find [7750833868800, 168, 262144]
> > >
> > > Oh it's data, interesting.  Pushed some more debugging.  Thanks,
> >
> > inserting block group 15840439173120
> > inserting block group 15842586656768
> > processed 1556480 of 0 possible bytes
> > processed 1130496 of 0 possible bytesdoing an insert that overlaps our bytenr 7750833627136 262144
> > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr??
> 
> Well crap, you have bytenrs that overlap.  I've adjusted
> init-extent-tree to dump the paths of the files that have overlapping
> extents, as well as the keys.  Run it again so we know what files are
> fucked. From there you need to tell me which one you don't care about
> and are willing to delete, and then I'll give you the command you need
> to remove that bad extent and then we can go again.  Thanks,

I see. 

Not sure if that helped. Should I have run it differently?


Author: Josef Bacik <josef@toxicpanda.com>
Date:   Wed Apr 27 16:19:46 2022 -0400
    fix some things so dump path works

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1                                                                                  
(...)
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0]
doing an insert that overlaps our bytenr 7750833627136 262144
processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths
Failed to find [7750833868800, 168, 262144]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffdce0, comp=comp@entry=0x55555559ae9a <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffdcf8) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffdce0, comp=comp@entry=0x55555559ae9a <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffdcf8) at common/rbtree-utils.c:48
#1  0x000055555559b09e in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=100564992) at common/extent-cache.c:179
#2  0x0000555555584c2c in set_extent_bits (tree=0x100000060, start=100564992, end=100564991, bits=bits@entry=1)
    at kernel-shared/extent_io.c:380
#3  0x0000555555584e5f in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x0000555555585842 in set_extent_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/extent_io.c:976
#5  0x000055555557bc10 in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee39 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998756304, parent=0, 
    iref=0xffffffffffffffe3, path=0x555555b8cfd0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998756304, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b8cfd0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555dfcf4 in process_eb (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d00d20, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:555
#10 0x00005555555dfe4d in process_eb (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, eb=0x555555944100, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:630
#11 0x00005555555e00b8 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:701
#12 0x00005555555e03c2 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:837
#13 0x00005555555d7a08 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffe3c0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#17 main (argc=3, argv=0x7fffffffe3c0) at btrfs.c:405


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-27 21:02                                                                                                                                                                                                                 ` Marc MERLIN
@ 2022-04-27 21:11                                                                                                                                                                                                                   ` Josef Bacik
  2022-04-27 21:20                                                                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-27 21:11 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 5:02 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Apr 27, 2022 at 04:21:31PM -0400, Josef Bacik wrote:
> > On Wed, Apr 27, 2022 at 2:24 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Wed, Apr 27, 2022 at 01:49:46PM -0400, Josef Bacik wrote:
> > > > > Result:
> > > > > doing insert of 12233379401728
> > > > > Failed to find [7750833868800, 168, 262144]
> > > >
> > > > Oh it's data, interesting.  Pushed some more debugging.  Thanks,
> > >
> > > inserting block group 15840439173120
> > > inserting block group 15842586656768
> > > processed 1556480 of 0 possible bytes
> > > processed 1130496 of 0 possible bytesdoing an insert that overlaps our bytenr 7750833627136 262144
> > > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr??
> >
> > Well crap, you have bytenrs that overlap.  I've adjusted
> > init-extent-tree to dump the paths of the files that have overlapping
> > extents, as well as the keys.  Run it again so we know what files are
> > fucked. From there you need to tell me which one you don't care about
> > and are willing to delete, and then I'll give you the command you need
> > to remove that bad extent and then we can go again.  Thanks,
>
> I see.
>
> Not sure if that helped. Should I have run it differently?
>
>
> Author: Josef Bacik <josef@toxicpanda.com>
> Date:   Wed Apr 27 16:19:46 2022 -0400
>     fix some things so dump path works
>
> (gdb) run rescue init-extent-tree /dev/mapper/dshelf1
> (...)
> inserting block group 15838291689472
> inserting block group 15839365431296
> inserting block group 15840439173120
> inserting block group 15842586656768
> processed 1556480 of 0 possible bytes
> processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0]
> doing an insert that overlaps our bytenr 7750833627136 262144
> processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths
> Failed to find [7750833868800, 168, 262144]
>

Of course it doesn't work for you, I pushed some debug stuff.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-27 21:11                                                                                                                                                                                                                   ` Josef Bacik
@ 2022-04-27 21:20                                                                                                                                                                                                                     ` Marc MERLIN
  2022-04-27 21:27                                                                                                                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-27 21:20 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 05:11:30PM -0400, Josef Bacik wrote:
> > inserting block group 15838291689472
> > inserting block group 15839365431296
> > inserting block group 15840439173120
> > inserting block group 15842586656768
> > processed 1556480 of 0 possible bytes
> > processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0]
> > doing an insert that overlaps our bytenr 7750833627136 262144
> > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths
> > Failed to find [7750833868800, 168, 262144]
> >
> 
> Of course it doesn't work for you, I pushed some debug stuff.  Thanks,

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
Couldn't find the last root for 8
checksum verify failed on 58720256 wanted 0x0d38525a found 0xb3a707fa
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks
(...)
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0]
elem_cnt 0 elem_missed 0 ret -2
doing an insert that overlaps our bytenr 7750833627136 262144
processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths
elem_cnt 0 elem_missed 0 ret -2
Failed to find [7750833868800, 168, 262144]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffdce0, comp=comp@entry=0x55555559ae9a <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffdcf8) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffdce0, comp=comp@entry=0x55555559ae9a <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffdcf8) at common/rbtree-utils.c:48
#1  0x000055555559b09e in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=100564992) at common/extent-cache.c:179
#2  0x0000555555584c2c in set_extent_bits (tree=0x100000060, start=100564992, end=100564991, bits=bits@entry=1)
    at kernel-shared/extent_io.c:380
#3  0x0000555555584e5f in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x0000555555585842 in set_extent_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/extent_io.c:976
#5  0x000055555557bc10 in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee39 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998755744, parent=0, 
    iref=0xffffffffffffffe3, path=0x555555b8cda0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998755744, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b8cda0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555558a8d6a0, root=root@entry=0x55555564cde0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555dfd17 in process_eb (trans=trans@entry=0x555558a8d6a0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d00f00, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:557
#10 0x00005555555dfe70 in process_eb (trans=trans@entry=0x555558a8d6a0, root=root@entry=0x55555564cde0, eb=0x555555944100, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:632
#11 0x00005555555e00db in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:703
#12 0x00005555555e03e5 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:839
#13 0x00005555555d7a08 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffe3c0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#17 main (argc=3, argv=0x7fffffffe3c0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-27 21:20                                                                                                                                                                                                                     ` Marc MERLIN
@ 2022-04-27 21:27                                                                                                                                                                                                                       ` Josef Bacik
  2022-04-27 22:59                                                                                                                                                                                                                         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-27 21:27 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 5:20 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Apr 27, 2022 at 05:11:30PM -0400, Josef Bacik wrote:
> > > inserting block group 15838291689472
> > > inserting block group 15839365431296
> > > inserting block group 15840439173120
> > > inserting block group 15842586656768
> > > processed 1556480 of 0 possible bytes
> > > processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0]
> > > doing an insert that overlaps our bytenr 7750833627136 262144
> > > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths
> > > Failed to find [7750833868800, 168, 262144]
> > >
> >
> > Of course it doesn't work for you, I pushed some debug stuff.  Thanks,
>
> (gdb) run rescue init-extent-tree /dev/mapper/dshelf1
> Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> FS_INFO IS 0x55555564cbc0
> JOSEF: root 9
> Couldn't find the last root for 8
> checksum verify failed on 58720256 wanted 0x0d38525a found 0xb3a707fa
> FS_INFO AFTER IS 0x55555564cbc0
> Walking all our trees and pinning down the currently accessible blocks
> (...)
> inserting block group 15842586656768
> processed 1556480 of 0 possible bytes
> processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0]
> elem_cnt 0 elem_missed 0 ret -2
> doing an insert that overlaps our bytenr 7750833627136 262144
> processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths
> elem_cnt 0 elem_missed 0 ret -2
> Failed to find [7750833868800, 168, 262144]

Sigh, added another print_leaf.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-27 21:27                                                                                                                                                                                                                       ` Josef Bacik
@ 2022-04-27 22:59                                                                                                                                                                                                                         ` Marc MERLIN
  2022-04-27 23:02                                                                                                                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-27 22:59 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 05:27:44PM -0400, Josef Bacik wrote:
> Sigh, added another print_leaf.  Thanks,

doing an insert that overlaps our bytenr 7750833627136 262144
processed 1146880 of 0 possible bytes
processed 1163264 of 0 possible bytes
processed 1179648 of 0 possible bytes
processed 1196032 of 0 possible bytes
processed 1212416 of 0 possible bytes
processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths
inode ref info failed???
leaf 15645023322112 items 123 free space 55 generation 1546750 owner ROOT_TREE
leaf 15645023322112 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
	item 0 key (5457 INODE_ITEM 0) itemoff 16123 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1835008000
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 7000 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.690807153 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 1 key (5457 EXTENT_DATA 0) itemoff 16070 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12207705702400 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 2 key (5458 INODE_ITEM 0) itemoff 15910 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 46137344
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 176 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.690807153 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 3 key (5458 EXTENT_DATA 0) itemoff 15857 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12207791230976 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 4 key (5459 INODE_ITEM 0) itemoff 15697 itemsize 160
		generation 1482901 transid 1482901 size 262144 nbytes 74448896
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 284 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603169214.790245505 (2020-10-19 21:46:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 5 key (5459 EXTENT_DATA 0) itemoff 15644 itemsize 53
		generation 1482901 type 1 (regular)
		extent data disk byte 7314895085568 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 6 key (5460 INODE_ITEM 0) itemoff 15484 itemsize 160
		generation 1439787 transid 1439787 size 262144 nbytes 398721024
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1521 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1591038874.596346316 (2020-06-01 12:14:34)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 7 key (5460 EXTENT_DATA 0) itemoff 15431 itemsize 53
		generation 1439787 type 1 (regular)
		extent data disk byte 12470912008192 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 8 key (5461 INODE_ITEM 0) itemoff 15271 itemsize 160
		generation 1482901 transid 1482901 size 262144 nbytes 25165824
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 96 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603169214.790245505 (2020-10-19 21:46:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 9 key (5461 EXTENT_DATA 0) itemoff 15218 itemsize 53
		generation 1482901 type 1 (regular)
		extent data disk byte 7482517606400 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 10 key (5462 INODE_ITEM 0) itemoff 15058 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1549795328
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5912 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.694807100 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 11 key (5462 EXTENT_DATA 0) itemoff 15005 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12207794606080 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 12 key (5463 INODE_ITEM 0) itemoff 14845 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 746848256
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2849 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.694807100 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 13 key (5463 EXTENT_DATA 0) itemoff 14792 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12207801716736 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 14 key (5464 INODE_ITEM 0) itemoff 14632 itemsize 160
		generation 1482589 transid 1482589 size 262144 nbytes 686555136
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2619 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603031991.590888741 (2020-10-18 07:39:51)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 15 key (5464 EXTENT_DATA 0) itemoff 14579 itemsize 53
		generation 1482589 type 1 (regular)
		extent data disk byte 952344715264 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 16 key (5465 INODE_ITEM 0) itemoff 14419 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1620574208
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 6182 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.698807046 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 17 key (5465 EXTENT_DATA 0) itemoff 14366 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12207808765952 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 18 key (5466 INODE_ITEM 0) itemoff 14206 itemsize 160
		generation 1511028 transid 1511028 size 262144 nbytes 777256960
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2965 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1610956842.625245815 (2021-01-18 00:00:42)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 19 key (5466 EXTENT_DATA 0) itemoff 14153 itemsize 53
		generation 1511028 type 1 (regular)
		extent data disk byte 12245962477568 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 20 key (5467 INODE_ITEM 0) itemoff 13993 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1741684736
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 6644 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.702806994 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 21 key (5467 EXTENT_DATA 0) itemoff 13940 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12207860899840 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 22 key (5468 INODE_ITEM 0) itemoff 13780 itemsize 160
		generation 1525970 transid 1525970 size 262144 nbytes 95944704
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 366 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222802.713761123 (2021-03-08 09:00:02)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 23 key (5468 EXTENT_DATA 0) itemoff 13727 itemsize 53
		generation 1525970 type 1 (regular)
		extent data disk byte 7437273346048 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 24 key (5469 INODE_ITEM 0) itemoff 13567 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 3301703680
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 12595 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.702806994 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 25 key (5469 EXTENT_DATA 0) itemoff 13514 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12207862214656 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 26 key (5470 INODE_ITEM 0) itemoff 13354 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 2621440
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 10 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.706806940 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 27 key (5470 EXTENT_DATA 0) itemoff 13301 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12208843333632 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 28 key (5471 INODE_ITEM 0) itemoff 13141 itemsize 160
		generation 1482591 transid 1482591 size 262144 nbytes 992477184
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3786 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603032110.61310221 (2020-10-18 07:41:50)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 29 key (5471 EXTENT_DATA 0) itemoff 13088 itemsize 53
		generation 1482591 type 1 (regular)
		extent data disk byte 3184741777408 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 30 key (5472 INODE_ITEM 0) itemoff 12928 itemsize 160
		generation 1525968 transid 1525968 size 262144 nbytes 35389440
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 135 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222730.746702497 (2021-03-08 08:58:50)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 31 key (5472 EXTENT_DATA 0) itemoff 12875 itemsize 53
		generation 1525968 type 1 (regular)
		extent data disk byte 8818662686720 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 32 key (5473 INODE_ITEM 0) itemoff 12715 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 2621440
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 10 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.706806940 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 33 key (5473 EXTENT_DATA 0) itemoff 12662 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12209607069696 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 34 key (5474 INODE_ITEM 0) itemoff 12502 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 521666560
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1990 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.710806887 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 35 key (5474 EXTENT_DATA 0) itemoff 12449 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12209981120512 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 36 key (5475 INODE_ITEM 0) itemoff 12289 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1441792000
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5500 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.710806887 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 37 key (5475 EXTENT_DATA 0) itemoff 12236 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12210260660224 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 38 key (5476 INODE_ITEM 0) itemoff 12076 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1366556672
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5213 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.710806887 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 39 key (5476 EXTENT_DATA 0) itemoff 12023 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12210686017536 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 40 key (5477 INODE_ITEM 0) itemoff 11863 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 7077888
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 27 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.714806834 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 41 key (5477 EXTENT_DATA 0) itemoff 11810 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12210843799552 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 42 key (5478 INODE_ITEM 0) itemoff 11650 itemsize 160
		generation 1482901 transid 1482901 size 262144 nbytes 2883584
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603169214.794245452 (2020-10-19 21:46:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 43 key (5478 EXTENT_DATA 0) itemoff 11597 itemsize 53
		generation 1482901 type 1 (regular)
		extent data disk byte 7513535868928 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 44 key (5479 INODE_ITEM 0) itemoff 11437 itemsize 160
		generation 1509233 transid 1509233 size 262144 nbytes 201850880
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 770 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1610527033.221370143 (2021-01-13 00:37:13)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 45 key (5479 EXTENT_DATA 0) itemoff 11384 itemsize 53
		generation 1509233 type 1 (regular)
		extent data disk byte 11682327523328 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 46 key (5480 INODE_ITEM 0) itemoff 11224 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 968884224
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3696 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.714806834 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 47 key (5480 EXTENT_DATA 0) itemoff 11171 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12210946199552 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 48 key (5481 INODE_ITEM 0) itemoff 11011 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1839464448
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 7017 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.718806781 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 49 key (5481 EXTENT_DATA 0) itemoff 10958 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12211111063552 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 50 key (5482 INODE_ITEM 0) itemoff 10798 itemsize 160
		generation 1525968 transid 1525968 size 262144 nbytes 1832124416
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 6989 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222730.746702497 (2021-03-08 08:58:50)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 51 key (5482 EXTENT_DATA 0) itemoff 10745 itemsize 53
		generation 1525968 type 1 (regular)
		extent data disk byte 8818664521728 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 52 key (5483 INODE_ITEM 0) itemoff 10585 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 6291456
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 24 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.718806781 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 53 key (5483 EXTENT_DATA 0) itemoff 10532 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12211529863168 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 54 key (5484 INODE_ITEM 0) itemoff 10372 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 3145728
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.722806727 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 55 key (5484 EXTENT_DATA 0) itemoff 10319 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12228941504512 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 56 key (5485 INODE_ITEM 0) itemoff 10159 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 370933760
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1415 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.722806727 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 57 key (5485 EXTENT_DATA 0) itemoff 10106 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12231089319936 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 58 key (5486 INODE_ITEM 0) itemoff 9946 itemsize 160
		generation 1525967 transid 1525967 size 262144 nbytes 1185415168
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4522 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222694.723170138 (2021-03-08 08:58:14)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 59 key (5486 EXTENT_DATA 0) itemoff 9893 itemsize 53
		generation 1525967 type 1 (regular)
		extent data disk byte 9425865584640 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 60 key (5487 INODE_ITEM 0) itemoff 9733 itemsize 160
		generation 1501750 transid 1501750 size 262144 nbytes 1346895872
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5138 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1609833640.497822971 (2021-01-05 00:00:40)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 61 key (5487 EXTENT_DATA 0) itemoff 9680 itemsize 53
		generation 1501750 type 1 (regular)
		extent data disk byte 7873777274880 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 62 key (5488 INODE_ITEM 0) itemoff 9520 itemsize 160
		generation 1530421 transid 1530421 size 262144 nbytes 1375207424
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5246 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1617139918.873813209 (2021-03-30 14:31:58)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 63 key (5488 EXTENT_DATA 0) itemoff 9467 itemsize 53
		generation 1530421 type 1 (regular)
		extent data disk byte 7441055866880 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 64 key (5489 INODE_ITEM 0) itemoff 9307 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 2621440
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 10 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.726806674 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 65 key (5489 EXTENT_DATA 0) itemoff 9254 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12231929507840 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 66 key (5490 INODE_ITEM 0) itemoff 9094 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 3932160
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 15 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.730806621 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 67 key (5490 EXTENT_DATA 0) itemoff 9041 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12232045236224 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 68 key (5491 INODE_ITEM 0) itemoff 8881 itemsize 160
		generation 1482911 transid 1482911 size 262144 nbytes 17301504
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 66 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603169591.357269745 (2020-10-19 21:53:11)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 69 key (5491 EXTENT_DATA 0) itemoff 8828 itemsize 53
		generation 1482911 type 1 (regular)
		extent data disk byte 10889918492672 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 70 key (5492 INODE_ITEM 0) itemoff 8668 itemsize 160
		generation 1482901 transid 1482901 size 262144 nbytes 26214400
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 100 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603169214.794245452 (2020-10-19 21:46:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 71 key (5492 EXTENT_DATA 0) itemoff 8615 itemsize 53
		generation 1482901 type 1 (regular)
		extent data disk byte 7515684741120 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 72 key (5493 INODE_ITEM 0) itemoff 8455 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 18087936
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 69 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.730806621 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 73 key (5493 EXTENT_DATA 0) itemoff 8402 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233142820864 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 74 key (5494 INODE_ITEM 0) itemoff 8242 itemsize 160
		generation 1439787 transid 1439787 size 262144 nbytes 54263808
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 207 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1591038874.720343072 (2020-06-01 12:14:34)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 75 key (5494 EXTENT_DATA 0) itemoff 8189 itemsize 53
		generation 1439787 type 1 (regular)
		extent data disk byte 12473955663872 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 76 key (5495 INODE_ITEM 0) itemoff 8029 itemsize 160
		generation 1482908 transid 1482908 size 262144 nbytes 2883584
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603169482.790704298 (2020-10-19 21:51:22)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 77 key (5495 EXTENT_DATA 0) itemoff 7976 itemsize 53
		generation 1482908 type 1 (regular)
		extent data disk byte 9382917087232 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 78 key (5496 INODE_ITEM 0) itemoff 7816 itemsize 160
		generation 1482593 transid 1482593 size 262144 nbytes 873725952
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3333 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603032208.635996917 (2020-10-18 07:43:28)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 79 key (5496 EXTENT_DATA 0) itemoff 7763 itemsize 53
		generation 1482593 type 1 (regular)
		extent data disk byte 4583295479808 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 80 key (5497 INODE_ITEM 0) itemoff 7603 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1531183104
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5841 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.734806568 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 81 key (5497 EXTENT_DATA 0) itemoff 7550 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233145966592 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 82 key (5498 INODE_ITEM 0) itemoff 7390 itemsize 160
		generation 1439787 transid 1439787 size 262144 nbytes 18612224
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 71 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1591038874.724342967 (2020-06-01 12:14:34)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 83 key (5498 EXTENT_DATA 0) itemoff 7337 itemsize 53
		generation 1439787 type 1 (regular)
		extent data disk byte 12473955926016 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 84 key (5499 INODE_ITEM 0) itemoff 7177 itemsize 160
		generation 1525968 transid 1525968 size 262144 nbytes 432013312
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1648 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222730.750702446 (2021-03-08 08:58:50)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 85 key (5499 EXTENT_DATA 0) itemoff 7124 itemsize 53
		generation 1525968 type 1 (regular)
		extent data disk byte 8818666881024 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 86 key (5500 INODE_ITEM 0) itemoff 6964 itemsize 160
		generation 1482593 transid 1482593 size 262144 nbytes 266600448
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1017 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603032208.639996865 (2020-10-18 07:43:28)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 87 key (5500 EXTENT_DATA 0) itemoff 6911 itemsize 53
		generation 1482593 type 1 (regular)
		extent data disk byte 4654161346560 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 88 key (5501 INODE_ITEM 0) itemoff 6751 itemsize 160
		generation 1525961 transid 1525961 size 262144 nbytes 1067450368
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4072 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222478.881970135 (2021-03-08 08:54:38)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 89 key (5501 EXTENT_DATA 0) itemoff 6698 itemsize 53
		generation 1525961 type 1 (regular)
		extent data disk byte 7982222049280 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 90 key (5502 INODE_ITEM 0) itemoff 6538 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 562561024
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2146 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.734806568 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 91 key (5502 EXTENT_DATA 0) itemoff 6485 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233186123776 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 92 key (5503 INODE_ITEM 0) itemoff 6325 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 3145728
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.734806568 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 93 key (5503 EXTENT_DATA 0) itemoff 6272 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233187991552 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 94 key (5504 INODE_ITEM 0) itemoff 6112 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1200095232
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4578 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.738806514 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 95 key (5504 EXTENT_DATA 0) itemoff 6059 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233257922560 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 96 key (5505 INODE_ITEM 0) itemoff 5899 itemsize 160
		generation 1525961 transid 1525961 size 262144 nbytes 71827456
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 274 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222478.885970083 (2021-03-08 08:54:38)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 97 key (5505 EXTENT_DATA 0) itemoff 5846 itemsize 53
		generation 1525961 type 1 (regular)
		extent data disk byte 8013364232192 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 98 key (5506 INODE_ITEM 0) itemoff 5686 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 4718592
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 18 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.738806514 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 99 key (5506 EXTENT_DATA 0) itemoff 5633 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233379401728 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 100 key (5507 INODE_ITEM 0) itemoff 5473 itemsize 160
		generation 1525959 transid 1525959 size 262144 nbytes 294649856
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1124 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222403.234950654 (2021-03-08 08:53:23)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 101 key (5507 EXTENT_DATA 0) itemoff 5420 itemsize 53
		generation 1525959 type 1 (regular)
		extent data disk byte 7750833868800 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 102 key (5508 INODE_ITEM 0) itemoff 5260 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 362807296
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1384 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.742806462 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 103 key (5508 EXTENT_DATA 0) itemoff 5207 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233610690560 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 104 key (5509 INODE_ITEM 0) itemoff 5047 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 766246912
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2923 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.742806462 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 105 key (5509 EXTENT_DATA 0) itemoff 4994 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233544069120 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 106 key (5510 INODE_ITEM 0) itemoff 4834 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1768685568
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 6747 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.746806408 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 107 key (5510 EXTENT_DATA 0) itemoff 4781 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233581916160 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 108 key (5511 INODE_ITEM 0) itemoff 4621 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 958398464
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3656 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.746806408 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 109 key (5511 EXTENT_DATA 0) itemoff 4568 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233725329408 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 110 key (5512 INODE_ITEM 0) itemoff 4408 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 408420352
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1558 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.750806355 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 111 key (5512 EXTENT_DATA 0) itemoff 4355 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233793339392 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 112 key (5513 INODE_ITEM 0) itemoff 4195 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1253834752
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4783 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.750806355 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 113 key (5513 EXTENT_DATA 0) itemoff 4142 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233785352192 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 114 key (5514 INODE_ITEM 0) itemoff 3982 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 8126464
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 31 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.754806301 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 115 key (5514 EXTENT_DATA 0) itemoff 3929 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233813143552 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 116 key (5515 INODE_ITEM 0) itemoff 3769 itemsize 160
		generation 1439787 transid 1439787 size 262144 nbytes 44826624
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 171 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1591038874.816340561 (2020-06-01 12:14:34)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 117 key (5515 EXTENT_DATA 0) itemoff 3716 itemsize 53
		generation 1439787 type 1 (regular)
		extent data disk byte 12474011467776 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 118 key (5516 INODE_ITEM 0) itemoff 3556 itemsize 160
		generation 1546750 transid 1546750 size 262144 nbytes 2405433344
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 9176 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1621321238.520473625 (2021-05-18 00:00:38)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 119 key (5516 EXTENT_DATA 0) itemoff 3503 itemsize 53
		generation 1546750 type 1 (regular)
		extent data disk byte 12251618267136 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 120 key (5517 INODE_ITEM 0) itemoff 3343 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 40370176
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 154 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.758806249 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 121 key (5517 EXTENT_DATA 0) itemoff 3290 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233872674816 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 122 key (5518 INODE_ITEM 0) itemoff 3130 itemsize 160
		generation 1525963 transid 1525963 size 262144 nbytes 57933824
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 221 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222550.521041172 (2021-03-08 08:55:50)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
elem_cnt 0 elem_missed 0 ret -2
Failed to find [7750833868800, 168, 262144]
Segmentation fault


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-27 22:59                                                                                                                                                                                                                         ` Marc MERLIN
@ 2022-04-27 23:02                                                                                                                                                                                                                           ` Josef Bacik
  2022-04-27 23:21                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-27 23:02 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 6:59 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Apr 27, 2022 at 05:27:44PM -0400, Josef Bacik wrote:
> > Sigh, added another print_leaf.  Thanks,
>
> doing an insert that overlaps our bytenr 7750833627136 262144
> processed 1146880 of 0 possible bytes
> processed 1163264 of 0 possible bytes
> processed 1179648 of 0 possible bytes
> processed 1196032 of 0 possible bytes
> processed 1212416 of 0 possible bytes
> processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths
> inode ref info failed???
> leaf 15645023322112 items 123 free space 55 generation 1546750 owner ROOT_TREE

Ooooh that explains it, it's the free space cache, that's perfect!
I'll get something wired up and let you know when it's ready.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-27 23:02                                                                                                                                                                                                                           ` Josef Bacik
@ 2022-04-27 23:21                                                                                                                                                                                                                             ` Josef Bacik
  2022-04-28  0:18                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-27 23:21 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 7:02 PM Josef Bacik <josef@toxicpanda.com> wrote:
>
> On Wed, Apr 27, 2022 at 6:59 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Wed, Apr 27, 2022 at 05:27:44PM -0400, Josef Bacik wrote:
> > > Sigh, added another print_leaf.  Thanks,
> >
> > doing an insert that overlaps our bytenr 7750833627136 262144
> > processed 1146880 of 0 possible bytes
> > processed 1163264 of 0 possible bytes
> > processed 1179648 of 0 possible bytes
> > processed 1196032 of 0 possible bytes
> > processed 1212416 of 0 possible bytes
> > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths
> > inode ref info failed???
> > leaf 15645023322112 items 123 free space 55 generation 1546750 owner ROOT_TREE
>
> Ooooh that explains it, it's the free space cache, that's perfect!
> I'll get something wired up and let you know when it's ready.  Thanks,
>

Ok, lets hope for better results this time.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-27 23:21                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-04-28  0:18                                                                                                                                                                                                                               ` Marc MERLIN
  2022-04-28  0:44                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-28  0:18 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 07:21:51PM -0400, Josef Bacik wrote:
> On Wed, Apr 27, 2022 at 7:02 PM Josef Bacik <josef@toxicpanda.com> wrote:
> >
> > On Wed, Apr 27, 2022 at 6:59 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Wed, Apr 27, 2022 at 05:27:44PM -0400, Josef Bacik wrote:
> > > > Sigh, added another print_leaf.  Thanks,
> > >
> > > doing an insert that overlaps our bytenr 7750833627136 262144
> > > processed 1146880 of 0 possible bytes
> > > processed 1163264 of 0 possible bytes
> > > processed 1179648 of 0 possible bytes
> > > processed 1196032 of 0 possible bytes
> > > processed 1212416 of 0 possible bytes
> > > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths
> > > inode ref info failed???
> > > leaf 15645023322112 items 123 free space 55 generation 1546750 owner ROOT_TREE
> >
> > Ooooh that explains it, it's the free space cache, that's perfect!
> > I'll get something wired up and let you know when it's ready.  Thanks,
> >
> 
> Ok, lets hope for better results this time.  Thanks,

        item 115 key (5514 EXTENT_DATA 0) itemoff 3929 itemsize 53
                generation 1526158 type 1 (regular)
                extent data disk byte 12233813143552 nr 262144
                extent data offset 0 nr 262144 ram 262144
                extent compression 0 (none)
        item 116 key (5515 INODE_ITEM 0) itemoff 3769 itemsize 160
                generation 1439787 transid 1439787 size 262144 nbytes 44826624
                block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
                sequence 171 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
                atime 0.0 (1969-12-31 16:00:00)
                ctime 1591038874.816340561 (2020-06-01 12:14:34)
                mtime 0.0 (1969-12-31 16:00:00)
                otime 0.0 (1969-12-31 16:00:00)
        item 117 key (5515 EXTENT_DATA 0) itemoff 3716 itemsize 53
                generation 1439787 type 1 (regular)
                extent data disk byte 12474011467776 nr 262144
                extent data offset 0 nr 262144 ram 262144
                extent compression 0 (none)
        item 118 key (5516 INODE_ITEM 0) itemoff 3556 itemsize 160
                generation 1546750 transid 1546750 size 262144 nbytes 2405433344
                block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
                sequence 9176 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
                atime 0.0 (1969-12-31 16:00:00)
                ctime 1621321238.520473625 (2021-05-18 00:00:38)
                mtime 0.0 (1969-12-31 16:00:00)
                otime 0.0 (1969-12-31 16:00:00)
        item 119 key (5516 EXTENT_DATA 0) itemoff 3503 itemsize 53
                generation 1546750 type 1 (regular)
                extent data disk byte 12251618267136 nr 262144
                extent data offset 0 nr 262144 ram 262144
                extent compression 0 (none)
        item 120 key (5517 INODE_ITEM 0) itemoff 3343 itemsize 160
                generation 1526158 transid 1526158 size 262144 nbytes 40370176
                block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
                sequence 154 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
                atime 0.0 (1969-12-31 16:00:00)
                ctime 1615276843.758806249 (2021-03-09 00:00:43)
                mtime 0.0 (1969-12-31 16:00:00)
                otime 0.0 (1969-12-31 16:00:00)
        item 121 key (5517 EXTENT_DATA 0) itemoff 3290 itemsize 53
                generation 1526158 type 1 (regular)
                extent data disk byte 12233872674816 nr 262144
                extent data offset 0 nr 262144 ram 262144
                extent compression 0 (none)
        item 122 key (5518 INODE_ITEM 0) itemoff 3130 itemsize 160
                generation 1525963 transid 1525963 size 262144 nbytes 57933824
                block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
                sequence 221 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
                atime 0.0 (1969-12-31 16:00:00)
                ctime 1615222550.521041172 (2021-03-08 08:55:50)
                mtime 0.0 (1969-12-31 16:00:00)
                otime 0.0 (1969-12-31 16:00:00)
elem_cnt 0 elem_missed 0 ret -2
Failed to find [7750833868800, 168, 262144]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd740, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd758) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd740, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd758) at common/rbtree-utils.c:48
#1  0x000055555559b0d9 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=13576826748928)
    at common/extent-cache.c:179
#2  0x0000555555584c67 in set_extent_bits (tree=0x100000060, start=13576826748928, end=13576826748927, bits=bits@entry=1)
    at kernel-shared/extent_io.c:380
#3  0x0000555555584e9a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x000055555558587d in set_extent_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/extent_io.c:976
#5  0x000055555557bc4b in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee74 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998761968, parent=0, 
    iref=0xffffffffffffffe3, path=0x555555b8e5f0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998761968, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b8e5f0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555dfd52 in process_eb (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d00d20, 
    current=current@entry=0x7fffffffdaf8) at cmds/rescue-init-extent-tree.c:615
#10 0x00005555555dfeab in process_eb (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, eb=0x555555944100, 
    current=current@entry=0x7fffffffdaf8) at cmds/rescue-init-extent-tree.c:690
#11 0x00005555555e0116 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:761
#12 0x00005555555e0436 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:897
#13 0x00005555555d7a43 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#17 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28  0:18                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-04-28  0:44                                                                                                                                                                                                                                 ` Josef Bacik
  2022-04-28  3:00                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-28  0:44 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 8:18 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Apr 27, 2022 at 07:21:51PM -0400, Josef Bacik wrote:
> > On Wed, Apr 27, 2022 at 7:02 PM Josef Bacik <josef@toxicpanda.com> wrote:
> > >
> > > On Wed, Apr 27, 2022 at 6:59 PM Marc MERLIN <marc@merlins.org> wrote:
> > > >
> > > > On Wed, Apr 27, 2022 at 05:27:44PM -0400, Josef Bacik wrote:
> > > > > Sigh, added another print_leaf.  Thanks,
> > > >
> > > > doing an insert that overlaps our bytenr 7750833627136 262144
> > > > processed 1146880 of 0 possible bytes
> > > > processed 1163264 of 0 possible bytes
> > > > processed 1179648 of 0 possible bytes
> > > > processed 1196032 of 0 possible bytes
> > > > processed 1212416 of 0 possible bytes
> > > > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths
> > > > inode ref info failed???
> > > > leaf 15645023322112 items 123 free space 55 generation 1546750 owner ROOT_TREE
> > >
> > > Ooooh that explains it, it's the free space cache, that's perfect!
> > > I'll get something wired up and let you know when it's ready.  Thanks,
> > >
> >
> > Ok, lets hope for better results this time.  Thanks,
>

Ok it should work now.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28  0:44                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-04-28  3:00                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-04-28  3:08                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-28  3:00 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 08:44:01PM -0400, Josef Bacik wrote:
> Ok it should work now.  Thanks,

Mmmh, it got worse, unfortunately

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1 
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff78e1f46 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff78e1f46 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000555555585950 in memmove (__len=<optimized out>, __src=<optimized out>, __dest=<optimized out>)
    at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:36
#2  memmove_extent_buffer (dst=dst@entry=0x555555b57f60, dst_offset=<optimized out>, src_offset=<optimized out>, len=<optimized out>)
    at kernel-shared/extent_io.c:1021
#3  0x000055555557753b in btrfs_del_items (trans=trans@entry=0x555558a8d460, root=root@entry=0x55555564cde0, 
    path=path@entry=0x7fffffffdc50, slot=slot@entry=36, nr=nr@entry=1) at ./kernel-shared/ctree.h:2387
#4  0x00005555555e0627 in clear_space_cache (block_group=<optimized out>, trans=0x555558a8d460) at cmds/rescue-init-extent-tree.c:116
#5  reset_block_groups (trans=0x555558a8d460) at cmds/rescue-init-extent-tree.c:307
#6  reinit_extent_tree (fs_info=0x7fffffffdc50) at cmds/rescue-init-extent-tree.c:451
#7  btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:922
#8  0x00005555555d7a43 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#9  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#10 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#11 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#12 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28  3:00                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-04-28  3:08                                                                                                                                                                                                                                     ` Josef Bacik
  2022-04-28  3:11                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-28  3:08 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 11:00 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Apr 27, 2022 at 08:44:01PM -0400, Josef Bacik wrote:
> > Ok it should work now.  Thanks,
>
> Mmmh, it got worse, unfortunately
>

Well not worse, just failed earlier because that's where the new code
is.  I think I just made a simple mistake, but I added some print'fs
just in case.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28  3:08                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-04-28  3:11                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-04-28  4:03                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-28  3:11 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 11:08:02PM -0400, Josef Bacik wrote:
> On Wed, Apr 27, 2022 at 11:00 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Wed, Apr 27, 2022 at 08:44:01PM -0400, Josef Bacik wrote:
> > > Ok it should work now.  Thanks,
> >
> > Mmmh, it got worse, unfortunately
> >
> 
> Well not worse, just failed earlier because that's where the new code
> is.  I think I just made a simple mistake, but I added some print'fs
> just in case.  Thanks,

That helped:
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 18446744073706422272 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 1707278336 of 16545742848 possible byteskernel-shared/extent-tree.c:1193: insert_inline_extent_backref: BUG_ON `owner < BTRFS_FIRST_FREE_OBJECTID` triggered, value 1
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x2975e)[0x55555557d75e]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_inc_extent_ref+0x138)[0x55555557ed77]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8be01)[0x5555555dfe01]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8beab)[0x5555555dfeab]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8beab)[0x5555555dfeab]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8c116)[0x5555555e0116]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_init_extent_tree+0xe96)[0x5555555e1089]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x83a43)[0x5555555d7a43]
/var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b]
/var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd]
/var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]

Program received signal SIGABRT, Aborted.
0x00007ffff78768a1 in raise () from /lib/x86_64-linux-gnu/libc.so.6

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28  3:11                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-04-28  4:03                                                                                                                                                                                                                                         ` Josef Bacik
  2022-04-28  4:12                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-28  4:03 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Apr 27, 2022 at 11:11 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Apr 27, 2022 at 11:08:02PM -0400, Josef Bacik wrote:
> > On Wed, Apr 27, 2022 at 11:00 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Wed, Apr 27, 2022 at 08:44:01PM -0400, Josef Bacik wrote:
> > > > Ok it should work now.  Thanks,
> > >
> > > Mmmh, it got worse, unfortunately
> > >
> >
> > Well not worse, just failed earlier because that's where the new code
> > is.  I think I just made a simple mistake, but I added some print'fs
> > just in case.  Thanks,
>
> That helped:
> inserting block group 15839365431296
> inserting block group 15840439173120
> inserting block group 15842586656768
> processed 1556480 of 0 possible bytes
> processed 1474560 of 18446744073706422272 possible bytes
> Recording extents for root 4
> processed 1032192 of 1064960 possible bytes
> Recording extents for root 5
> processed 10960896 of 10977280 possible bytes
> Recording extents for root 7
> processed 1707278336 of 16545742848 possible byteskernel-shared/extent-tree.c:1193: insert_inline_extent_backref: BUG_ON `owner < BTRFS_FIRST_FREE_OBJECTID` triggered, value 1
> /var/local/src/btrfs-progs-josefbacik/btrfs(+0x2975e)[0x55555557d75e]
> /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_inc_extent_ref+0x138)[0x55555557ed77]
> /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8be01)[0x5555555dfe01]
> /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8beab)[0x5555555dfeab]
> /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8beab)[0x5555555dfeab]
> /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8c116)[0x5555555e0116]
> /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_init_extent_tree+0xe96)[0x5555555e1089]
> /var/local/src/btrfs-progs-josefbacik/btrfs(+0x83a43)[0x5555555d7a43]
> /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b]
> /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd]
> /var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]
>

Cool, we're pointing at the same block in different places in the same
tree.  I should make tree-recover catch that and fix it, but since
you're going to re-generate your csum tree anyway I've adjusted
init-extent-tree to just clear the csum tree too, lets see how far we
get with this.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28  4:03                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-04-28  4:12                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-04-28 15:30                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-28  4:12 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 12:03:29AM -0400, Josef Bacik wrote:
> Cool, we're pointing at the same block in different places in the same
> tree.  I should make tree-recover catch that and fix it, but since
> you're going to re-generate your csum tree anyway I've adjusted
> init-extent-tree to just clear the csum tree too, lets see how far we
> get with this.  Thanks,

inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1634992128 of 1635549184 possible bytesFailed to find [3700677820416, 168, 53248]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48
#1  0x000055555559b0d9 in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=15645286318080)
    at common/extent-cache.c:179
#2  0x0000555555584c67 in set_extent_bits (tree=0x10000000060, start=15645286318080, end=15645286318079, bits=bits@entry=1)
    at kernel-shared/extent_io.c:380
#3  0x0000555555584e9a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x000055555558587d in set_extent_buffer_dirty (eb=eb@entry=0x555560e15ef0) at kernel-shared/extent_io.c:976
#5  0x000055555557bc4b in btrfs_mark_buffer_dirty (eb=eb@entry=0x555560e15ef0) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee74 in setup_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825185865744, 
    parent=0, iref=0xffffffffffffffe4, path=0x555560dfe010, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825185865744, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555560dfe010, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=11223, owner=1834097, offset=1835008)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555dfea9 in process_eb (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, eb=eb@entry=0x555560e0ddf0, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:659
#10 0x00005555555e0002 in process_eb (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, eb=eb@entry=0x55555a1be9a0, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#11 0x00005555555e0002 in process_eb (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, eb=0x555559ce9020, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#12 0x00005555555e026d in record_root (root=root@entry=0x555555af1ed0) at cmds/rescue-init-extent-tree.c:805
#13 0x00005555555e10f9 in record_roots (fs_info=0x55555564cbc0) at cmds/rescue-init-extent-tree.c:860
#14 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:944
#15 0x00005555555d7a43 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28  4:12                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-04-28 15:30                                                                                                                                                                                                                                             ` Josef Bacik
  2022-04-28 16:27                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-28 15:30 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 12:12 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Apr 28, 2022 at 12:03:29AM -0400, Josef Bacik wrote:
> > Cool, we're pointing at the same block in different places in the same
> > tree.  I should make tree-recover catch that and fix it, but since
> > you're going to re-generate your csum tree anyway I've adjusted
> > init-extent-tree to just clear the csum tree too, lets see how far we
> > get with this.  Thanks,
>
> inserting block group 15835070464000
> inserting block group 15836144205824
> inserting block group 15837217947648
> inserting block group 15838291689472
> inserting block group 15839365431296
> inserting block group 15840439173120
> inserting block group 15842586656768
> processed 1556480 of 0 possible bytes
> processed 1474560 of 0 possible bytes
> Recording extents for root 4
> processed 1032192 of 1064960 possible bytes
> Recording extents for root 5
> processed 10960896 of 10977280 possible bytes
> Recording extents for root 7
> processed 16384 of 16545742848 possible bytes
> Recording extents for root 9
> processed 16384 of 16384 possible bytes
> Recording extents for root 11221
> processed 16384 of 255983616 possible bytes
> Recording extents for root 11222
> processed 49479680 of 49479680 possible bytes
> Recording extents for root 11223
> processed 1634992128 of 1635549184 possible bytesFailed to find [3700677820416, 168, 53248]
>

Hell yes we're in the fs tree's now, in the home stretch hopefully.
I've pushed new debugging, you may have another overlapping extent.
I'm going to have to wire up a tool for that, but hopefully we can
just target delete a few things and get you up and running.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28 15:30                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-04-28 16:27                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-04-28 20:13                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-28 16:27 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 11:30:35AM -0400, Josef Bacik wrote:
> Hell yes we're in the fs tree's now, in the home stretch hopefully.
> I've pushed new debugging, you may have another overlapping extent.
> I'm going to have to wire up a tool for that, but hopefully we can
> just target delete a few things and get you up and running.  Thanks,

Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ?

        item 186 key (1834097 EXTENT_DATA 56098816) itemoff 5452 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451680858112 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 187 key (1834097 EXTENT_DATA 56623104) itemoff 5399 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451683733504 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 188 key (1834097 EXTENT_DATA 57147392) itemoff 5346 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451684257792 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 189 key (1834097 EXTENT_DATA 57671680) itemoff 5293 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451684782080 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 190 key (1834097 EXTENT_DATA 58195968) itemoff 5240 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451685568512 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 191 key (1834097 EXTENT_DATA 58720256) itemoff 5187 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457553899520 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 192 key (1834097 EXTENT_DATA 59244544) itemoff 5134 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457554423808 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 193 key (1834097 EXTENT_DATA 59768832) itemoff 5081 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457561710592 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 194 key (1834097 EXTENT_DATA 60293120) itemoff 5028 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457563021312 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 195 key (1834097 EXTENT_DATA 60817408) itemoff 4975 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457564856320 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules
Failed to find [3700677820416, 168, 53248]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48
#1  0x000055555559b0d9 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645157752832)
    at common/extent-cache.c:179
#2  0x0000555555584c67 in set_extent_bits (tree=0x100000060, start=15645157752832, end=15645157752831, bits=bits@entry=1)
    at kernel-shared/extent_io.c:380
#3  0x0000555555584e9a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x000055555558587d in set_extent_buffer_dirty (eb=eb@entry=0x55555d2ba350) at kernel-shared/extent_io.c:976
#5  0x000055555557bc4b in btrfs_mark_buffer_dirty (eb=eb@entry=0x55555d2ba350) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee74 in setup_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123492528, 
    parent=0, iref=0xffffffffffffffe3, path=0x55555d2822b0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123492528, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x55555d2822b0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=11223, owner=1834097, offset=1835008)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555dfea9 in process_eb (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, eb=eb@entry=0x55555d291e50, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:659
#10 0x00005555555e0002 in process_eb (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, eb=eb@entry=0x5555564687c0, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#11 0x00005555555e0002 in process_eb (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, eb=0x555555f13e20, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#12 0x00005555555e026d in record_root (root=root@entry=0x555557a97ca0) at cmds/rescue-init-extent-tree.c:805
#13 0x00005555555e10f9 in record_roots (fs_info=0x55555564cbc0) at cmds/rescue-init-extent-tree.c:860
#14 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:944
#15 0x00005555555d7a43 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28 16:27                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-04-28 20:13                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-04-28 20:22                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-28 20:13 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 12:27 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Apr 28, 2022 at 11:30:35AM -0400, Josef Bacik wrote:
> > Hell yes we're in the fs tree's now, in the home stretch hopefully.
> > I've pushed new debugging, you may have another overlapping extent.
> > I'm going to have to wire up a tool for that, but hopefully we can
> > just target delete a few things and get you up and running.  Thanks,
>
> Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ?

Cool, do

./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 <device>

Then you should be able to run the init-extent-tree and get past that
part.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28 20:13                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-04-28 20:22                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-04-28 20:28                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-28 20:22 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 04:13:31PM -0400, Josef Bacik wrote:
> On Thu, Apr 28, 2022 at 12:27 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Thu, Apr 28, 2022 at 11:30:35AM -0400, Josef Bacik wrote:
> > > Hell yes we're in the fs tree's now, in the home stretch hopefully.
> > > I've pushed new debugging, you may have another overlapping extent.
> > > I'm going to have to wire up a tool for that, but hopefully we can
> > > just target delete a few things and get you up and running.  Thanks,
> >
> > Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ?
> 
> Cool, do
> 
> ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 <device>
> 
> Then you should be able to run the init-extent-tree and get past that
> part.  Thanks,

Sorry, I must be triggering every single unhandled path in the code :)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x55fd6e82d600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55fd6e82d600
kernel-shared/ctree.c:365: update_ref_for_cow: BUG_ON `ret` triggered, value -5
./btrfs-corrupt-block(+0x1137d)[0x55fd6d53237d]
./btrfs-corrupt-block(__btrfs_cow_block+0x3a9)[0x55fd6d533a90]
./btrfs-corrupt-block(btrfs_cow_block+0x73)[0x55fd6d53414f]
./btrfs-corrupt-block(btrfs_search_slot+0x11a)[0x55fd6d536b51]
./btrfs-corrupt-block(main+0x141d)[0x55fd6d56f4a9]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f526bee17fd]
./btrfs-corrupt-block(_start+0x2a)[0x55fd6d52d8ba]
Aborted

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28 20:22                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-04-28 20:28                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-04-28 20:57                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-28 20:28 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 4:22 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Apr 28, 2022 at 04:13:31PM -0400, Josef Bacik wrote:
> > On Thu, Apr 28, 2022 at 12:27 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Thu, Apr 28, 2022 at 11:30:35AM -0400, Josef Bacik wrote:
> > > > Hell yes we're in the fs tree's now, in the home stretch hopefully.
> > > > I've pushed new debugging, you may have another overlapping extent.
> > > > I'm going to have to wire up a tool for that, but hopefully we can
> > > > just target delete a few things and get you up and running.  Thanks,
> > >
> > > Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ?
> >
> > Cool, do
> >
> > ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 <device>
> >
> > Then you should be able to run the init-extent-tree and get past that
> > part.  Thanks,
>
> Sorry, I must be triggering every single unhandled path in the code :)
>

"Make recovery tools more resilient" is definitely muuuuuch higher on
the team priority list that's for fucking sure.  Try again please,
hopefully it works this time.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28 20:28                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-04-28 20:57                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-04-28 20:58                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-28 20:57 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 04:28:53PM -0400, Josef Bacik wrote:
> "Make recovery tools more resilient" is definitely muuuuuch higher on
> the team priority list that's for fucking sure.  Try again please,
> hopefully it works this time.  Thanks,

Great to hear, and obviously not just hearing, thanks for your
persistence in this mega thread.

Here's the new one:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d
"3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x55f248b64600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55f248b64600
Error searching to node -2

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28 20:57                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-04-28 20:58                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-04-28 21:42                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-28 20:58 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 4:57 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Apr 28, 2022 at 04:28:53PM -0400, Josef Bacik wrote:
> > "Make recovery tools more resilient" is definitely muuuuuch higher on
> > the team priority list that's for fucking sure.  Try again please,
> > hopefully it works this time.  Thanks,
>
> Great to hear, and obviously not just hearing, thanks for your
> persistence in this mega thread.
>
> Here's the new one:
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d
> "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1
> FS_INFO IS 0x55f248b64600
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x55f248b64600
> Error searching to node -2
>

-2 is enoent, it must have committed with the deleted block, which is
sort of scary but at this point I'll take it.  Go ahead and do the
init-extent-tree.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28 20:58                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-04-28 21:42                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-04-28 21:54                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-28 21:42 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 04:58:24PM -0400, Josef Bacik wrote:
> -2 is enoent, it must have committed with the deleted block, which is
> sort of scary but at this point I'll take it.  Go ahead and do the
> init-extent-tree.  Thanks,

Same output :(
Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules
Failed to find [3700677820416, 168, 53248]

it's the one we just tried to clear, and got enoent on.
Unless -r 11223 is not the correct value? (not sure where it came from)
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x55d8bac70600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55d8bac70600
Error searching to node -2

New run still returns the same thing:
        item 188 key (1834097 EXTENT_DATA 57147392) itemoff 5346 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451684257792 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 189 key (1834097 EXTENT_DATA 57671680) itemoff 5293 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451684782080 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 190 key (1834097 EXTENT_DATA 58195968) itemoff 5240 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451685568512 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 191 key (1834097 EXTENT_DATA 58720256) itemoff 5187 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457553899520 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 192 key (1834097 EXTENT_DATA 59244544) itemoff 5134 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457554423808 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 193 key (1834097 EXTENT_DATA 59768832) itemoff 5081 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457561710592 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 194 key (1834097 EXTENT_DATA 60293120) itemoff 5028 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457563021312 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 195 key (1834097 EXTENT_DATA 60817408) itemoff 4975 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457564856320 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules
Failed to find [3700677820416, 168, 53248]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aee0 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aee0 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48
#1  0x000055555559b0e4 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645157801984)
    at common/extent-cache.c:179
#2  0x0000555555584c72 in set_extent_bits (tree=0x100000060, start=15645157801984, end=15645157801983, bits=bits@entry=1)
    at kernel-shared/extent_io.c:380
#3  0x0000555555584ea5 in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x0000555555585888 in set_extent_buffer_dirty (eb=eb@entry=0x55555d2ab870) at kernel-shared/extent_io.c:976
#5  0x000055555557bc56 in btrfs_mark_buffer_dirty (eb=eb@entry=0x55555d2ab870) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee7f in setup_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123431232, 
    parent=0, iref=0xffffffffffffffe3, path=0x55555d273340, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123431232, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x55555d273340, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=11223, owner=1834097, offset=1835008)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555dfeb4 in process_eb (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, eb=eb@entry=0x55555d283370, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:659
#10 0x00005555555e000d in process_eb (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, eb=eb@entry=0x555556461260, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#11 0x00005555555e000d in process_eb (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, eb=0x555555f0c680, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#12 0x00005555555e0278 in record_root (root=root@entry=0x555556faefe0) at cmds/rescue-init-extent-tree.c:805
#13 0x00005555555e1104 in record_roots (fs_info=0x55555564cbc0) at cmds/rescue-init-extent-tree.c:860
#14 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:944
#15 0x00005555555d7a4e in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28 21:42                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-04-28 21:54                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-04-28 22:27                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-28 21:54 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 5:42 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Apr 28, 2022 at 04:58:24PM -0400, Josef Bacik wrote:
> > -2 is enoent, it must have committed with the deleted block, which is
> > sort of scary but at this point I'll take it.  Go ahead and do the
> > init-extent-tree.  Thanks,
>
> Same output :(
> Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules
> Failed to find [3700677820416, 168, 53248]
>
> it's the one we just tried to clear, and got enoent on.
> Unless -r 11223 is not the correct value? (not sure where it came from)
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1
> FS_INFO IS 0x55d8bac70600
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x55d8bac70600
> Error searching to node -2
>

Oooh I'm stupid, I thought the key it was printing out was the extent
we needed to delete, but it's the extent in the extent tree.  You cut
off the part I need, but that's because I'm printing the leaf when I
don't need to.

I've fixed the output so it should print out something like

[number, 108, number] dumping paths

that's what you want to feed into btrfs-corrupt-block, that should
delete the problematic item and then we can continue.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28 21:54                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-04-28 22:27                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-04-28 23:24                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-28 22:27 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 05:54:46PM -0400, Josef Bacik wrote:
> Oooh I'm stupid, I thought the key it was printing out was the extent
> we needed to delete, but it's the extent in the extent tree.  You cut
> off the part I need, but that's because I'm printing the leaf when I
> don't need to.
> 
> I've fixed the output so it should print out something like
> 
> [number, 108, number] dumping paths
> 
> that's what you want to feed into btrfs-corrupt-block, that should
> delete the problematic item and then we can continue.  Thanks,

inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 49152 of 0 possible bytesadding a bytenr that overlaps our
thing, dumping paths for [4088, 108, 0]
inode ref info failed???
elem_cnt 0 elem_missed 0 ret -2
doing an insert of the bytenr
doing an insert that overlaps our bytenr 3700677820416 262144
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1634992128 of 1635549184 possible bytesWTF???? we think we
already inserted this bytenr?? [1834097, 108, 1835008] dumping paths
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules
Failed to find [3700677820416, 168, 53248]
Segmentation fault

Isn't that root 11223 and
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1

why does it say root 11223 does not exists?
FS_INFO IS 0x558818fe3600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x558818fe3600
parent transid verify failed on 13576823668736 wanted 1619060 found
1619070
parent transid verify failed on 13576823668736 wanted 1619060 found
1619070
couldn't find root 11223


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28 22:27                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-04-28 23:24                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-04-29  0:56                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-28 23:24 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 6:27 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Apr 28, 2022 at 05:54:46PM -0400, Josef Bacik wrote:
> > Oooh I'm stupid, I thought the key it was printing out was the extent
> > we needed to delete, but it's the extent in the extent tree.  You cut
> > off the part I need, but that's because I'm printing the leaf when I
> > don't need to.
> >
> > I've fixed the output so it should print out something like
> >
> > [number, 108, number] dumping paths
> >
> > that's what you want to feed into btrfs-corrupt-block, that should
> > delete the problematic item and then we can continue.  Thanks,
>
> inserting block group 15835070464000
> inserting block group 15836144205824
> inserting block group 15837217947648
> inserting block group 15838291689472
> inserting block group 15839365431296
> inserting block group 15840439173120
> inserting block group 15842586656768
> processed 1556480 of 0 possible bytes
> processed 49152 of 0 possible bytesadding a bytenr that overlaps our
> thing, dumping paths for [4088, 108, 0]

Oh huh, we must not have a free space object for this, in that case lets do

./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/whatever

and then do the init.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-28 23:24                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-04-29  0:56                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-04-29  1:11                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-29  0:56 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 07:24:22PM -0400, Josef Bacik wrote:
> > inserting block group 15835070464000
> > inserting block group 15836144205824
> > inserting block group 15837217947648
> > inserting block group 15838291689472
> > inserting block group 15839365431296
> > inserting block group 15840439173120
> > inserting block group 15842586656768
> > processed 1556480 of 0 possible bytes
> > processed 49152 of 0 possible bytesadding a bytenr that overlaps our
> > thing, dumping paths for [4088, 108, 0]
> 
> Oh huh, we must not have a free space object for this, in that case lets do
> 
> ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/whatever
> 
> and then do the init.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/mapper/dshelf1
FS_INFO IS 0x558c0e536600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x558c0e536600
Error searching to node -2

not good news I assume?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-29  0:56                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-04-29  1:11                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-04-29  1:34                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-29  1:11 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 8:56 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Apr 28, 2022 at 07:24:22PM -0400, Josef Bacik wrote:
> > > inserting block group 15835070464000
> > > inserting block group 15836144205824
> > > inserting block group 15837217947648
> > > inserting block group 15838291689472
> > > inserting block group 15839365431296
> > > inserting block group 15840439173120
> > > inserting block group 15842586656768
> > > processed 1556480 of 0 possible bytes
> > > processed 49152 of 0 possible bytesadding a bytenr that overlaps our
> > > thing, dumping paths for [4088, 108, 0]
> >
> > Oh huh, we must not have a free space object for this, in that case lets do
> >
> > ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/whatever
> >
> > and then do the init.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/mapper/dshelf1
> FS_INFO IS 0x558c0e536600
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x558c0e536600
> Error searching to node -2
>
> not good news I assume?
>

Just that I´m dumb and making silly mistakes, its -r 1, sorry about that,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-29  1:11                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-04-29  1:34                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-04-29  1:38                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-29  1:34 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 09:11:51PM -0400, Josef Bacik wrote:
> On Thu, Apr 28, 2022 at 8:56 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Thu, Apr 28, 2022 at 07:24:22PM -0400, Josef Bacik wrote:
> > > > inserting block group 15835070464000
> > > > inserting block group 15836144205824
> > > > inserting block group 15837217947648
> > > > inserting block group 15838291689472
> > > > inserting block group 15839365431296
> > > > inserting block group 15840439173120
> > > > inserting block group 15842586656768
> > > > processed 1556480 of 0 possible bytes
> > > > processed 49152 of 0 possible bytesadding a bytenr that overlaps our
> > > > thing, dumping paths for [4088, 108, 0]
> > >
> > > Oh huh, we must not have a free space object for this, in that case lets do
> > >
> > > ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/whatever
> > >
> > > and then do the init.  Thanks,
> >
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/mapper/dshelf1
> > FS_INFO IS 0x558c0e536600
> > JOSEF: root 9
> > Couldn't find the last root for 8
> > FS_INFO AFTER IS 0x558c0e536600
> > Error searching to node -2
> >
> > not good news I assume?
> >
> 
> Just that I´m dumb and making silly mistakes, its -r 1, sorry about that,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 1 /dev/mapper/dshelf1
FS_INFO IS 0x557f68dd6600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x557f68dd6600

then init-extent-tree:
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
ERROR: root [11223 4061] level 0 does not match 2

ERROR: Error loading root
it failed?? -5
Init extent tree failed
[Inferior 1 (process 31788) exited with code 0373]

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-29  1:34                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-04-29  1:38                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-04-29  4:03                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-29  1:38 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 9:34 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Apr 28, 2022 at 09:11:51PM -0400, Josef Bacik wrote:
> > On Thu, Apr 28, 2022 at 8:56 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Thu, Apr 28, 2022 at 07:24:22PM -0400, Josef Bacik wrote:
> > > > > inserting block group 15835070464000
> > > > > inserting block group 15836144205824
> > > > > inserting block group 15837217947648
> > > > > inserting block group 15838291689472
> > > > > inserting block group 15839365431296
> > > > > inserting block group 15840439173120
> > > > > inserting block group 15842586656768
> > > > > processed 1556480 of 0 possible bytes
> > > > > processed 49152 of 0 possible bytesadding a bytenr that overlaps our
> > > > > thing, dumping paths for [4088, 108, 0]
> > > >
> > > > Oh huh, we must not have a free space object for this, in that case lets do
> > > >
> > > > ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/whatever
> > > >
> > > > and then do the init.  Thanks,
> > >
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/mapper/dshelf1
> > > FS_INFO IS 0x558c0e536600
> > > JOSEF: root 9
> > > Couldn't find the last root for 8
> > > FS_INFO AFTER IS 0x558c0e536600
> > > Error searching to node -2
> > >
> > > not good news I assume?
> > >
> >
> > Just that I´m dumb and making silly mistakes, its -r 1, sorry about that,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 1 /dev/mapper/dshelf1
> FS_INFO IS 0x557f68dd6600
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x557f68dd6600
>
> then init-extent-tree:
> inserting block group 15838291689472
> inserting block group 15839365431296
> inserting block group 15840439173120
> inserting block group 15842586656768
> processed 1556480 of 0 possible bytes
> processed 1474560 of 0 possible bytes
> Recording extents for root 4
> processed 1032192 of 1064960 possible bytes
> Recording extents for root 5
> processed 10960896 of 10977280 possible bytes
> Recording extents for root 7
> processed 16384 of 16545742848 possible bytes
> Recording extents for root 9
> processed 16384 of 16384 possible bytes
> Recording extents for root 11221
> processed 16384 of 255983616 possible bytes
> Recording extents for root 11222
> processed 49479680 of 49479680 possible bytes
> Ignoring transid failure
> ERROR: root [11223 4061] level 0 does not match 2
>

I'm going to scream.  Somehow the root pointer for 11223 got messed up
in all of this, do rescue tree-recover again so it can unfuck 11223,
and then init-extent-tree.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-29  1:38                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-04-29  4:03                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-04-29 12:41                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-29  4:03 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Apr 28, 2022 at 09:38:05PM -0400, Josef Bacik wrote:
> I'm going to scream.  Somehow the root pointer for 11223 got messed up
> in all of this, do rescue tree-recover again so it can unfuck 11223,
> and then init-extent-tree.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x5588db830bc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5588db830bc0
Checking root 2 bytenr 15645018570752
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 15645018554368
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736
Checking root 11222 bytenr 15645261905920
Checking root 11223 bytenr 13576823668736
Repairing root 11223 bad_blocks 9 update 1
deleting slot 54 in block 11652138631168
deleting slot 54 in block 11652138631168
deleting slot 64 in block 11652138631168
deleting slot 86 in block 11652138631168
deleting slot 87 in block 11652138631168
deleting slot 87 in block 11652138631168
deleting slot 87 in block 11652138631168
deleting slot 87 in block 11652138631168
deleting slot 87 in block 11652138631168
Checking root 11224 bytenr 13577126182912
Checking root 159785 bytenr 6781490577408
Checking root 159787 bytenr 15645908385792
Checking root 160494 bytenr 6781491265536
Checking root 160496 bytenr 11822309965824
Checking root 161197 bytenr 6781492101120
Checking root 161199 bytenr 13576850833408
Checking root 162628 bytenr 15645764812800
Checking root 162632 bytenr 6781492756480
Checking root 162645 bytenr 5809981095936
Checking root 163298 bytenr 15645124263936
Checking root 163302 bytenr 6781495197696
Checking root 163303 bytenr 15645365993472
Checking root 163316 bytenr 6781496393728
Checking root 163318 bytenr 15645980491776
Checking root 163916 bytenr 11822437826560
Checking root 163920 bytenr 11971021275136
Checking root 163921 bytenr 11971073802240
Checking root 164620 bytenr 15645434036224
Checking root 164624 bytenr 15645502210048
Checking root 164633 bytenr 15645526884352
Checking root 165098 bytenr 11970667446272
Checking root 165100 bytenr 11970733621248
Checking root 165198 bytenr 12511656394752
Checking root 165200 bytenr 12511677972480
Checking root 165294 bytenr 13576901328896
Checking root 165298 bytenr 13577133326336
Checking root 165299 bytenr 13577191505920
Checking root 18446744073709551607 bytenr 13576823799808
Tree recovery finished, you can run check now


gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1634844672 of 1635549184 possible bytesadding a bytenr that overlaps our thing, dumping paths for [1834097, 108, 1835008]
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules
doing an insert of the bytenr
doing an insert that overlaps our bytenr 3700677820416 53248
processed 1635319808 of 1635549184 possible bytes
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 999424 of 109035520 possible bytesFailed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555564fce1f0, parent_transid=parent_transid@entry=1619126)
    at kernel-shared/disk-io.c:2235
2235            ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,
(gdb) bt
#0  0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555564fce1f0, parent_transid=parent_transid@entry=1619126)
    at kernel-shared/disk-io.c:2235
#1  0x000055555557bebd in read_tree_block (fs_info=fs_info@entry=0x55555564cbc0, bytenr=<optimized out>, parent_transid=1619126)
    at kernel-shared/disk-io.c:378
#2  0x00005555555735f6 in read_node_slot (fs_info=fs_info@entry=0x55555564cbc0, root=root@entry=0x5555559547d0, 
    parent=0x555559e7e7b0, slot=240) at ./kernel-shared/ctree.h:1941
#3  0x00005555555769ff in btrfs_search_slot (trans=0x5555599539a0, root=root@entry=0x5555559547d0, key=key@entry=0x7fffffffd8a0, 
    p=p@entry=0x7fffffffd760, ins_len=ins_len@entry=49, cow=cow@entry=1) at kernel-shared/ctree.c:1429
#4  0x0000555555576e91 in btrfs_insert_empty_items (trans=trans@entry=0x5555599539a0, root=root@entry=0x5555559547d0, 
    path=path@entry=0x7fffffffd760, cpu_key=cpu_key@entry=0x7fffffffd8a0, data_size=data_size@entry=0x7fffffffd75c, nr=nr@entry=1)
    at kernel-shared/ctree.c:2824
#5  0x00005555555dfb9d in btrfs_insert_empty_item (data_size=<optimized out>, key=0x7fffffffd8a0, path=0x7fffffffd760, 
    root=0x5555559547d0, trans=0x5555599539a0) at ./kernel-shared/ctree.h:2780
#6  insert_empty_extent (trans=trans@entry=0x5555599539a0, key=key@entry=0x7fffffffd8a0, generation=generation@entry=1591667, 
    flags=flags@entry=1) at cmds/rescue-init-extent-tree.c:566
#7  0x00005555555dfe62 in process_eb (trans=trans@entry=0x5555599539a0, root=root@entry=0x555562939390, eb=eb@entry=0x555559092800, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:645
#8  0x00005555555dffed in process_eb (trans=trans@entry=0x5555599539a0, root=root@entry=0x555562939390, eb=eb@entry=0x5555590aaf80, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#9  0x00005555555dffed in process_eb (trans=trans@entry=0x5555599539a0, root=root@entry=0x555562939390, eb=0x5555590822a0, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#10 0x00005555555e0258 in record_root (root=root@entry=0x555562939390) at cmds/rescue-init-extent-tree.c:805
#11 0x00005555555e10e4 in record_roots (fs_info=0x55555564cbc0) at cmds/rescue-init-extent-tree.c:860
#12 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:944
#13 0x00005555555d7a2e in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#17 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-29  4:03                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-04-29 12:41                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-04-29 15:16                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-29 12:41 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Fri, Apr 29, 2022 at 12:03 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Apr 28, 2022 at 09:38:05PM -0400, Josef Bacik wrote:
> > I'm going to scream.  Somehow the root pointer for 11223 got messed up
> > in all of this, do rescue tree-recover again so it can unfuck 11223,
> > and then init-extent-tree.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> FS_INFO IS 0x5588db830bc0
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x5588db830bc0
> Checking root 2 bytenr 15645018570752
> Checking root 4 bytenr 15645196861440
> Checking root 5 bytenr 13577660252160
> Checking root 7 bytenr 15645018554368
> Checking root 9 bytenr 15645878108160
> Checking root 11221 bytenr 13577562996736
> Checking root 11222 bytenr 15645261905920
> Checking root 11223 bytenr 13576823668736
> Repairing root 11223 bad_blocks 9 update 1
> deleting slot 54 in block 11652138631168
> deleting slot 54 in block 11652138631168
> deleting slot 64 in block 11652138631168
> deleting slot 86 in block 11652138631168
> deleting slot 87 in block 11652138631168
> deleting slot 87 in block 11652138631168
> deleting slot 87 in block 11652138631168
> deleting slot 87 in block 11652138631168
> deleting slot 87 in block 11652138631168
> Checking root 11224 bytenr 13577126182912
> Checking root 159785 bytenr 6781490577408
> Checking root 159787 bytenr 15645908385792
> Checking root 160494 bytenr 6781491265536
> Checking root 160496 bytenr 11822309965824
> Checking root 161197 bytenr 6781492101120
> Checking root 161199 bytenr 13576850833408
> Checking root 162628 bytenr 15645764812800
> Checking root 162632 bytenr 6781492756480
> Checking root 162645 bytenr 5809981095936
> Checking root 163298 bytenr 15645124263936
> Checking root 163302 bytenr 6781495197696
> Checking root 163303 bytenr 15645365993472
> Checking root 163316 bytenr 6781496393728
> Checking root 163318 bytenr 15645980491776
> Checking root 163916 bytenr 11822437826560
> Checking root 163920 bytenr 11971021275136
> Checking root 163921 bytenr 11971073802240
> Checking root 164620 bytenr 15645434036224
> Checking root 164624 bytenr 15645502210048
> Checking root 164633 bytenr 15645526884352
> Checking root 165098 bytenr 11970667446272
> Checking root 165100 bytenr 11970733621248
> Checking root 165198 bytenr 12511656394752
> Checking root 165200 bytenr 12511677972480
> Checking root 165294 bytenr 13576901328896
> Checking root 165298 bytenr 13577133326336
> Checking root 165299 bytenr 13577191505920
> Checking root 18446744073709551607 bytenr 13576823799808
> Tree recovery finished, you can run check now
>
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
> inserting block group 15838291689472
> inserting block group 15839365431296
> inserting block group 15840439173120
> inserting block group 15842586656768
> processed 1556480 of 0 possible bytes
> processed 1474560 of 0 possible bytes
> Recording extents for root 4
> processed 1032192 of 1064960 possible bytes
> Recording extents for root 5
> processed 10960896 of 10977280 possible bytes
> Recording extents for root 7
> processed 16384 of 16545742848 possible bytes
> Recording extents for root 9
> processed 16384 of 16384 possible bytes
> Recording extents for root 11221
> processed 16384 of 255983616 possible bytes
> Recording extents for root 11222
> processed 49479680 of 49479680 possible bytes
> Recording extents for root 11223
> processed 1634844672 of 1635549184 possible bytesadding a bytenr that overlaps our thing, dumping paths for [1834097, 108, 1835008]
> inode ref info failed???
> elem_cnt 1 elem_missed 0 ret 0
> Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules
> doing an insert of the bytenr
> doing an insert that overlaps our bytenr 3700677820416 53248
> processed 1635319808 of 1635549184 possible bytes
> Recording extents for root 11224
> processed 75792384 of 75792384 possible bytes
> Recording extents for root 159785
> processed 108855296 of 108855296 possible bytes
> Recording extents for root 159787
> processed 49152 of 49479680 possible bytes
> Recording extents for root 160494
> processed 999424 of 109035520 possible bytesFailed to find [10467695652864, 168, 8675328]

Yayyyy progress, I just updated the debugging for the new problem
bytenr so we can figure out who to delete, run init-extent-tree again
please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-29 12:41                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-04-29 15:16                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-04-29 15:27                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-29 15:16 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Fri, Apr 29, 2022 at 08:41:00AM -0400, Josef Bacik wrote:
> Yayyyy progress, I just updated the debugging for the new problem
> bytenr so we can figure out who to delete, run init-extent-tree again
> please.  Thanks,

sure:
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 999424 of 109035520 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555565529a60, parent_transid=parent_transid@entry=1619139)
    at kernel-shared/disk-io.c:2235
2235            ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,

We're making progress, thanks. Is there a way to add a BUG_ON instead of
the subsequent SEGV?

I'm also not too sure, how to you convert  [10467695652864, 168, 8675328]
into something like
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 1 /dev/mapper/dshelf1

It seems like the straight numbers don't work:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "10467695652864,168,8675328" -r 1 /dev/mapper/dshelf1
FS_INFO IS 0x55e92004b600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55e92004b600
Error searching to node -2

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-29 15:16                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-04-29 15:27                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-04-29 17:16                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-29 15:27 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Fri, Apr 29, 2022 at 11:16 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Fri, Apr 29, 2022 at 08:41:00AM -0400, Josef Bacik wrote:
> > Yayyyy progress, I just updated the debugging for the new problem
> > bytenr so we can figure out who to delete, run init-extent-tree again
> > please.  Thanks,
>
> sure:
> inserting block group 15837217947648
> inserting block group 15838291689472
> inserting block group 15839365431296
> inserting block group 15840439173120
> inserting block group 15842586656768
> processed 1556480 of 0 possible bytes
> processed 1474560 of 0 possible bytes
> Recording extents for root 4
> processed 1032192 of 1064960 possible bytes
> Recording extents for root 5
> processed 10960896 of 10977280 possible bytes
> Recording extents for root 7
> processed 16384 of 16545742848 possible bytes
> Recording extents for root 9
> processed 16384 of 16384 possible bytes
> Recording extents for root 11221
> processed 16384 of 255983616 possible bytes
> Recording extents for root 11222
> processed 49479680 of 49479680 possible bytes
> Recording extents for root 11223
> processed 1635319808 of 1635549184 possible bytes
> Recording extents for root 11224
> processed 75792384 of 75792384 possible bytes
> Recording extents for root 159785
> processed 108855296 of 108855296 possible bytes
> Recording extents for root 159787
> processed 49152 of 49479680 possible bytes
> Recording extents for root 160494
> processed 999424 of 109035520 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths
> inode ref info failed???
> elem_cnt 1 elem_missed 0 ret 0
> misc/add0/new/tv1/ST/testfile.avi
> Failed to find [10467695652864, 168, 8675328]
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555565529a60, parent_transid=parent_transid@entry=1619139)
>     at kernel-shared/disk-io.c:2235
> 2235            ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,
>
> We're making progress, thanks. Is there a way to add a BUG_ON instead of
> the subsequent SEGV?

Yup done.  Also it's because I'm looking for the key in "dumping
paths" higher up.  This one is weird because we only got one output,
so for this one you want

-d "76300,108,0" -r 160494

Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-29 15:27                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-04-29 17:16                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-04-29 17:52                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-29 17:16 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Fri, Apr 29, 2022 at 11:27:54AM -0400, Josef Bacik wrote:
> > processed 999424 of 109035520 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths
> > inode ref info failed???
> > elem_cnt 1 elem_missed 0 ret 0
> > misc/add0/new/tv1/ST/testfile.avi
> > Failed to find [10467695652864, 168, 8675328]
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > 0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555565529a60, parent_transid=parent_transid@entry=1619139)
> >     at kernel-shared/disk-io.c:2235
> > 2235            ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,
> >
> > We're making progress, thanks. Is there a way to add a BUG_ON instead of
> > the subsequent SEGV?
> 
> Yup done.  Also it's because I'm looking for the key in "dumping
> paths" higher up.  This one is weird because we only got one output,
> so for this one you want
> 
> -d "76300,108,0" -r 160494

Not sure how you got those numbers, but sure. I think it worked, but it
doesn't give any output saying so
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "76300,108,0" -r 160494 /dev/mapper/dshelf1
FS_INFO IS 0x5563243d9600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5563243d9600

not sure it worked:
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 1179648 of 109035520 possible bytes
Recording extents for root 160496
processed 49152 of 49479680 possible bytes
Recording extents for root 161197
processed 81920 of 109019136 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x5555611a6d20, parent_transid=parent_transid@entry=1619156)
    at kernel-shared/disk-io.c:2235
2235            ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-29 17:16                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-04-29 17:52                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-04-29 18:58                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-29 17:52 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Fri, Apr 29, 2022 at 1:16 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Fri, Apr 29, 2022 at 11:27:54AM -0400, Josef Bacik wrote:
> > > processed 999424 of 109035520 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths
> > > inode ref info failed???
> > > elem_cnt 1 elem_missed 0 ret 0
> > > misc/add0/new/tv1/ST/testfile.avi
> > > Failed to find [10467695652864, 168, 8675328]
> > >
> > > Program received signal SIGSEGV, Segmentation fault.
> > > 0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555565529a60, parent_transid=parent_transid@entry=1619139)
> > >     at kernel-shared/disk-io.c:2235
> > > 2235            ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,
> > >
> > > We're making progress, thanks. Is there a way to add a BUG_ON instead of
> > > the subsequent SEGV?
> >
> > Yup done.  Also it's because I'm looking for the key in "dumping
> > paths" higher up.  This one is weird because we only got one output,
> > so for this one you want
> >
> > -d "76300,108,0" -r 160494
>
> Not sure how you got those numbers, but sure. I think it worked, but it
> doesn't give any output saying so
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "76300,108,0" -r 160494 /dev/mapper/dshelf1
> FS_INFO IS 0x5563243d9600
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x5563243d9600
>
> not sure it worked:
> inserting block group 15840439173120
> inserting block group 15842586656768
> processed 1556480 of 0 possible bytes
> processed 1474560 of 0 possible bytes
> Recording extents for root 4
> processed 1032192 of 1064960 possible bytes
> Recording extents for root 5
> processed 10960896 of 10977280 possible bytes
> Recording extents for root 7
> processed 16384 of 16545742848 possible bytes
> Recording extents for root 9
> processed 16384 of 16384 possible bytes
> Recording extents for root 11221
> processed 16384 of 255983616 possible bytes
> Recording extents for root 11222
> processed 49479680 of 49479680 possible bytes
> Recording extents for root 11223
> processed 1635319808 of 1635549184 possible bytes
> Recording extents for root 11224
> processed 75792384 of 75792384 possible bytes
> Recording extents for root 159785
> processed 108855296 of 108855296 possible bytes
> Recording extents for root 159787
> processed 49152 of 49479680 possible bytes
> Recording extents for root 160494
> processed 1179648 of 109035520 possible bytes
> Recording extents for root 160496
> processed 49152 of 49479680 possible bytes
> Recording extents for root 161197
> processed 81920 of 109019136 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths
> inode ref info failed???
> elem_cnt 1 elem_missed 0 ret 0
> misc/add0/new/tv1/ST/testfile.avi
> Failed to find [10467695652864, 168, 8675328]

It did, now we're on a different root, you can do

btrfs-corrupt-block -d "76300,108,0" -r 161197 <device>

And keep it rolling.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-29 17:52                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-04-29 18:58                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-04-29 19:40                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-29 18:58 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Fri, Apr 29, 2022 at 01:52:13PM -0400, Josef Bacik wrote:
> It did, now we're on a different root, you can do
> 
> btrfs-corrupt-block -d "76300,108,0" -r 161197 <device>

So, it looks like this is going to be long manual process, I went through iterations and 
keep getting new roots. When I clear one, it goes to the next.
Any way to automate this?

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "76300,108,0" -r 161197 /dev/mapper/dshelf1
FS_INFO IS 0x55be3db38600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55be3db38600

Still breaking on the same file, 
(...)
Recording extents for root 162632 <-------------------------------------------------------------------------
processed 1523712 of 109314048 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555562149b30, parent_transid=parent_transid@entry=1619175)
    at kernel-shared/disk-io.c:2235
2235            ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "76300,108,0" -r 162632 /dev/mapper/dshelf1
FS_INFO IS 0x55a45a034600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55a45a034600


And then same thing again next round:
processed 49152 of 49479680 possible bytes
Recording extents for root 163302 <---------------------------------------------------------------------------
processed 32768 of 109314048 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x55555fbab9e0, parent_transid=parent_transid@entry=1619197)
    at kernel-shared/disk-io.c:2235
2235            ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "76300,108,0" -r 163303 /dev/mapper/dshelf1
FS_INFO IS 0x564c9f705600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x564c9f705600

processed 81920 of 75792384 possible bytes
Recording extents for root 163316 <---------------------------------------------------------------------------
WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x5555608f4060, parent_transid=parent_transid@entry=1619222)
    at kernel-shared/disk-io.c:2235
2235            ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-29 18:58                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-04-29 19:40                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-04-30  2:24                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-29 19:40 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Fri, Apr 29, 2022 at 2:58 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Fri, Apr 29, 2022 at 01:52:13PM -0400, Josef Bacik wrote:
> > It did, now we're on a different root, you can do
> >
> > btrfs-corrupt-block -d "76300,108,0" -r 161197 <device>
>
> So, it looks like this is going to be long manual process, I went through iterations and
> keep getting new roots. When I clear one, it goes to the next.
> Any way to automate this?
>

I'm afraid it'll be longer/less safe for me to work out the kinks than
to continue manually removing stuff.  If you have to do it say 10 more
times let me know and I can try and rig up something that can dump
root ids that you can just feed into btrfs-corrupt-block to clear
everything out at once.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-29 19:40                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-04-30  2:24                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-04-30  3:13                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-30  2:24 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Fri, Apr 29, 2022 at 03:40:53PM -0400, Josef Bacik wrote:
> 
> I'm afraid it'll be longer/less safe for me to work out the kinks than
> to continue manually removing stuff.  If you have to do it say 10 more
> times let me know and I can try and rig up something that can dump
> root ids that you can just feed into btrfs-corrupt-block to clear
> everything out at once.  Thanks,

Took a while:
./btrfs-corrupt-block -d "76300,108,0" -r 163302 /dev/mapper/dshelf1
./btrfs-corrupt-block -d "76300,108,0" -r 163303 /dev/mapper/dshelf1
./btrfs-corrupt-block -d "76300,108,0" -r 163316 /dev/mapper/dshelf1
./btrfs-corrupt-block -d "76300,108,0" -r 163920 /dev/mapper/dshelf1
./btrfs-corrupt-block -d "76300,108,0" -r 164624 /dev/mapper/dshelf1
./btrfs-corrupt-block -d "76300,108,0" -r 165098 /dev/mapper/dshelf1
./btrfs-corrupt-block -d "76300,108,0" -r 165198 /dev/mapper/dshelf1
./btrfs-corrupt-block -d "76300,108,0" -r 165298 /dev/mapper/dshelf1
./btrfs-corrupt-block -d "76300,108,0" -r 16384 /dev/mapper/dshelf1



Now I'm stuck here:

processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13576823734272 len 16384
Init extent tree failed
[Inferior 1 (process 31377) exited with code 0377]

./btrfs-corrupt-block -d "76300,108,0" -r 18446744073709551607 /dev/mapper/dshelf1

Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13576823734272 len 16384
Init extent tree failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-30  2:24                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-04-30  3:13                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-04-30 13:07                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-30  3:13 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Fri, Apr 29, 2022 at 10:24 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Fri, Apr 29, 2022 at 03:40:53PM -0400, Josef Bacik wrote:
> >
> > I'm afraid it'll be longer/less safe for me to work out the kinks than
> > to continue manually removing stuff.  If you have to do it say 10 more
> > times let me know and I can try and rig up something that can dump
> > root ids that you can just feed into btrfs-corrupt-block to clear
> > everything out at once.  Thanks,
>
> Took a while:
> ./btrfs-corrupt-block -d "76300,108,0" -r 163302 /dev/mapper/dshelf1
> ./btrfs-corrupt-block -d "76300,108,0" -r 163303 /dev/mapper/dshelf1
> ./btrfs-corrupt-block -d "76300,108,0" -r 163316 /dev/mapper/dshelf1
> ./btrfs-corrupt-block -d "76300,108,0" -r 163920 /dev/mapper/dshelf1
> ./btrfs-corrupt-block -d "76300,108,0" -r 164624 /dev/mapper/dshelf1
> ./btrfs-corrupt-block -d "76300,108,0" -r 165098 /dev/mapper/dshelf1
> ./btrfs-corrupt-block -d "76300,108,0" -r 165198 /dev/mapper/dshelf1
> ./btrfs-corrupt-block -d "76300,108,0" -r 165298 /dev/mapper/dshelf1
> ./btrfs-corrupt-block -d "76300,108,0" -r 16384 /dev/mapper/dshelf1
>
>
>
> Now I'm stuck here:
>

Hooray we're at the data reloc root.  It should have been cleared tho,
so I've fixed it up to see if it's doing the right thing, it should
clear it this time, if it doesn't let me know.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-30  3:13                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-04-30 13:07                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-04-30 16:40                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-30 13:07 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Fri, Apr 29, 2022 at 11:13:08PM -0400, Josef Bacik wrote:
> Hooray we're at the data reloc root.  It should have been cleared tho,
> so I've fixed it up to see if it's doing the right thing, it should
> clear it this time, if it doesn't let me know.  Thanks,

Great

FYI:
cmds/rescue-init-extent-tree.c:417:64: warning: passing argument 1 of ‘PTR_ERR’ makes pointer from integer without a cast [-Wint-conversion]
  417 |   fprintf(stderr, "Error reading data reloc tree %d\n",PTR_ERR(ret));
      |                                                                ^~~
      |                                                                |
      |                                                                int
In file included from cmds/rescue-init-extent-tree.c:3:
./kerncompat.h:270:40: note: expected ‘const void *’ but argument is of type ‘int’
  270 | static inline long PTR_ERR(const void *ptr)
      |                            ~~~~~~~~~~~~^~~
cmds/rescue-init-extent-tree.c:417:51: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘long int’ [-Wformat=]
  417 |   fprintf(stderr, "Error reading data reloc tree %d\n",PTR_ERR(ret));
      |                                                  ~^    ~~~~~~~~~~~~
      |                                                   |    |
      |                                                   int  long int
      |                                                  %ld

(..)
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
DID THE DATA RELOC TREE INIT?!
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 1179648 of 109035520 possible bytes
Recording extents for root 160496
processed 49152 of 49479680 possible bytes
Recording extents for root 161197
processed 147456 of 109019136 possible bytes
Recording extents for root 161199
processed 49152 of 49479680 possible bytes
Recording extents for root 162628
processed 49152 of 49479680 possible bytes
Recording extents for root 162632
processed 2129920 of 109314048 possible bytes
Recording extents for root 162645
processed 49152 of 75792384 possible bytes
Recording extents for root 163298
processed 49152 of 49479680 possible bytes
Recording extents for root 163302
processed 147456 of 109314048 possible bytes
Recording extents for root 163303
processed 81920 of 75792384 possible bytes
Recording extents for root 163316
processed 49152 of 109314048 possible bytes
Recording extents for root 163318
processed 16384 of 49479680 possible bytes
Recording extents for root 163916
processed 49152 of 49479680 possible bytes
Recording extents for root 163920
processed 81920 of 109314048 possible bytes
Recording extents for root 163921
processed 49152 of 75792384 possible bytes
Recording extents for root 164620
processed 49152 of 49479680 possible bytes
Recording extents for root 164624
processed 491520 of 109445120 possible bytes
Recording extents for root 164633
processed 49152 of 75792384 possible bytes
Recording extents for root 165098
processed 212992 of 109445120 possible bytes
Recording extents for root 165100
processed 16384 of 49479680 possible bytes
Recording extents for root 165198
processed 49152 of 109445120 possible bytes
Recording extents for root 165200
processed 16384 of 49479680 possible bytes
Recording extents for root 165294
processed 16384 of 49479680 possible bytes
Recording extents for root 165298
processed 81920 of 109445120 possible bytes
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13576823652352 len 16384
Init extent tree failed
[Inferior 1 (process 9144) exited with code 0377]
(gdb) 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-30 13:07                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-04-30 16:40                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-04-30 23:11                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-04-30 16:40 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sat, Apr 30, 2022 at 9:07 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Fri, Apr 29, 2022 at 11:13:08PM -0400, Josef Bacik wrote:
> > Hooray we're at the data reloc root.  It should have been cleared tho,
> > so I've fixed it up to see if it's doing the right thing, it should
> > clear it this time, if it doesn't let me know.  Thanks,
>
> Great
>

Ok this is easier, the transaction commit stuff doesn't work quite
right in progs, I've fixed the code and it should succeed this time.
Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-30 16:40                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-04-30 23:11                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-05-01  2:48                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-04-30 23:11 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sat, Apr 30, 2022 at 12:40:46PM -0400, Josef Bacik wrote:
> On Sat, Apr 30, 2022 at 9:07 AM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Fri, Apr 29, 2022 at 11:13:08PM -0400, Josef Bacik wrote:
> > > Hooray we're at the data reloc root.  It should have been cleared tho,
> > > so I've fixed it up to see if it's doing the right thing, it should
> > > clear it this time, if it doesn't let me know.  Thanks,
> >
> > Great
> >
> 
> Ok this is easier, the transaction commit stuff doesn't work quite
> right in progs, I've fixed the code and it should succeed this time.
> Thanks,


ainserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 1179648 of 109035520 possible bytes
Recording extents for root 160496
processed 49152 of 49479680 possible bytes
Recording extents for root 161197
processed 147456 of 109019136 possible bytes
Recording extents for root 161199
processed 49152 of 49479680 possible bytes
Recording extents for root 162628
processed 49152 of 49479680 possible bytes
Recording extents for root 162632
processed 2129920 of 109314048 possible bytes
Recording extents for root 162645
processed 49152 of 75792384 possible bytes
Recording extents for root 163298
processed 49152 of 49479680 possible bytes
Recording extents for root 163302
processed 147456 of 109314048 possible bytes
Recording extents for root 163303
processed 81920 of 75792384 possible bytes
Recording extents for root 163316
processed 49152 of 109314048 possible bytes
Recording extents for root 163318
processed 16384 of 49479680 possible bytes
Recording extents for root 163916
processed 49152 of 49479680 possible bytes
Recording extents for root 163920
processed 81920 of 109314048 possible bytes
Recording extents for root 163921
processed 49152 of 75792384 possible bytes
Recording extents for root 164620
processed 49152 of 49479680 possible bytes
Recording extents for root 164624
processed 491520 of 109445120 possible bytes
Recording extents for root 164633
processed 49152 of 75792384 possible bytes
Recording extents for root 165098
processed 212992 of 109445120 possible bytes
Recording extents for root 165100
processed 16384 of 49479680 possible bytes
Recording extents for root 165198
processed 49152 of 109445120 possible bytes
Recording extents for root 165200
processed 16384 of 49479680 possible bytes
Recording extents for root 165294
processed 16384 of 49479680 possible bytes
Recording extents for root 165298
processed 81920 of 109445120 possible bytes
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13576823832576 len 16384
Init extent tree failed
[Inferior 1 (process 28313) exited with code 0377]

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-04-30 23:11                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-05-01  2:48                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-05-01  4:54                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-01  2:48 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sat, Apr 30, 2022 at 7:11 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sat, Apr 30, 2022 at 12:40:46PM -0400, Josef Bacik wrote:
> > On Sat, Apr 30, 2022 at 9:07 AM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Fri, Apr 29, 2022 at 11:13:08PM -0400, Josef Bacik wrote:
> > > > Hooray we're at the data reloc root.  It should have been cleared tho,
> > > > so I've fixed it up to see if it's doing the right thing, it should
> > > > clear it this time, if it doesn't let me know.  Thanks,
> > >
> > > Great
> > >
> >
> > Ok this is easier, the transaction commit stuff doesn't work quite
> > right in progs, I've fixed the code and it should succeed this time.
> > Thanks,
>
>
> ainserting block group 15837217947648
> inserting block group 15838291689472
> inserting block group 15839365431296
> inserting block group 15840439173120
> inserting block group 15842586656768
> processed 1556480 of 0 possible bytes
> processed 1474560 of 0 possible bytes
> Recording extents for root 4
> processed 1032192 of 1064960 possible bytes
> Recording extents for root 5
> processed 10960896 of 10977280 possible bytes
> Recording extents for root 7
> processed 16384 of 16545742848 possible bytes
> Recording extents for root 9
> processed 16384 of 16384 possible bytes
> Recording extents for root 11221
> processed 16384 of 255983616 possible bytes
> Recording extents for root 11222
> processed 49479680 of 49479680 possible bytes
> Recording extents for root 11223
> processed 1635319808 of 1635549184 possible bytes
> Recording extents for root 11224
> processed 75792384 of 75792384 possible bytes
> Recording extents for root 159785
> processed 108855296 of 108855296 possible bytes
> Recording extents for root 159787
> processed 49152 of 49479680 possible bytes
> Recording extents for root 160494
> processed 1179648 of 109035520 possible bytes
> Recording extents for root 160496
> processed 49152 of 49479680 possible bytes
> Recording extents for root 161197
> processed 147456 of 109019136 possible bytes
> Recording extents for root 161199
> processed 49152 of 49479680 possible bytes
> Recording extents for root 162628
> processed 49152 of 49479680 possible bytes
> Recording extents for root 162632
> processed 2129920 of 109314048 possible bytes
> Recording extents for root 162645
> processed 49152 of 75792384 possible bytes
> Recording extents for root 163298
> processed 49152 of 49479680 possible bytes
> Recording extents for root 163302
> processed 147456 of 109314048 possible bytes
> Recording extents for root 163303
> processed 81920 of 75792384 possible bytes
> Recording extents for root 163316
> processed 49152 of 109314048 possible bytes
> Recording extents for root 163318
> processed 16384 of 49479680 possible bytes
> Recording extents for root 163916
> processed 49152 of 49479680 possible bytes
> Recording extents for root 163920
> processed 81920 of 109314048 possible bytes
> Recording extents for root 163921
> processed 49152 of 75792384 possible bytes
> Recording extents for root 164620
> processed 49152 of 49479680 possible bytes
> Recording extents for root 164624
> processed 491520 of 109445120 possible bytes
> Recording extents for root 164633
> processed 49152 of 75792384 possible bytes
> Recording extents for root 165098
> processed 212992 of 109445120 possible bytes
> Recording extents for root 165100
> processed 16384 of 49479680 possible bytes
> Recording extents for root 165198
> processed 49152 of 109445120 possible bytes
> Recording extents for root 165200
> processed 16384 of 49479680 possible bytes
> Recording extents for root 165294
> processed 16384 of 49479680 possible bytes
> Recording extents for root 165298
> processed 81920 of 109445120 possible bytes
> Recording extents for root 165299
> processed 16384 of 75792384 possible bytes
> Recording extents for root 18446744073709551607
> processed 16384 of 16384 possible bytes
> ERROR: commit_root already set when starting transaction

Well it looks like it finished, but I don't see my "start transaction
failed" messages which should be printing, I've added some extra
debugging to figure out wherever this is failing.  We're literally
done and of course it's failing somewhere at the end, hopefully
this'll be quicker to nail down.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-01  2:48                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-05-01  4:54                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-05-01 11:28                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-01  4:54 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sat, Apr 30, 2022 at 10:48:08PM -0400, Josef Bacik wrote:
> > Recording extents for root 165299
> > processed 16384 of 75792384 possible bytes
> > Recording extents for root 18446744073709551607
> > processed 16384 of 16384 possible bytes
> > ERROR: commit_root already set when starting transaction
> 
> Well it looks like it finished, but I don't see my "start transaction
> failed" messages which should be printing, I've added some extra
> debugging to figure out wherever this is failing.  We're literally
> done and of course it's failing somewhere at the end, hopefully
> this'll be quicker to nail down.  Thanks,

Good news :)
Here is the new one:
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 1179648 of 109035520 possible bytes
Recording extents for root 160496
processed 49152 of 49479680 possible bytes
Recording extents for root 161197
processed 147456 of 109019136 possible bytes
Recording extents for root 161199
processed 49152 of 49479680 possible bytes
Recording extents for root 162628
processed 49152 of 49479680 possible bytes
Recording extents for root 162632
processed 2129920 of 109314048 possible bytes
Recording extents for root 162645
processed 49152 of 75792384 possible bytes
Recording extents for root 163298
processed 49152 of 49479680 possible bytes
Recording extents for root 163302
processed 147456 of 109314048 possible bytes
Recording extents for root 163303
processed 81920 of 75792384 possible bytes
Recording extents for root 163316
processed 49152 of 109314048 possible bytes
Recording extents for root 163318
processed 16384 of 49479680 possible bytes
Recording extents for root 163916
processed 49152 of 49479680 possible bytes
Recording extents for root 163920
processed 81920 of 109314048 possible bytes
Recording extents for root 163921
processed 49152 of 75792384 possible bytes
Recording extents for root 164620
processed 49152 of 49479680 possible bytes
Recording extents for root 164624
processed 491520 of 109445120 possible bytes
Recording extents for root 164633
processed 49152 of 75792384 possible bytes
Recording extents for root 165098
processed 212992 of 109445120 possible bytes
Recording extents for root 165100
processed 16384 of 49479680 possible bytes
Recording extents for root 165198
processed 49152 of 109445120 possible bytes
Recording extents for root 165200
processed 16384 of 49479680 possible bytes
Recording extents for root 165294
processed 16384 of 49479680 possible bytes
Recording extents for root 165298
processed 81920 of 109445120 possible bytes
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
doing close???
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13576823750656 len 16384
Init extent tree failed
[Inferior 1 (process 8777) exited with code 0377]

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-01  4:54                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-05-01 11:28                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-05-01 15:22                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-01 11:28 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 1, 2022 at 12:54 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sat, Apr 30, 2022 at 10:48:08PM -0400, Josef Bacik wrote:
> > > Recording extents for root 165299
> > > processed 16384 of 75792384 possible bytes
> > > Recording extents for root 18446744073709551607
> > > processed 16384 of 16384 possible bytes
> > > ERROR: commit_root already set when starting transaction
> >
> > Well it looks like it finished, but I don't see my "start transaction
> > failed" messages which should be printing, I've added some extra
> > debugging to figure out wherever this is failing.  We're literally
> > done and of course it's failing somewhere at the end, hopefully
> > this'll be quicker to nail down.  Thanks,
>
> Good news :)
> Here is the new one:
> doing roots
> Recording extents for root 4
> processed 1032192 of 1064960 possible bytes
> Recording extents for root 5
> processed 10960896 of 10977280 possible bytes
> Recording extents for root 7
> processed 16384 of 16545742848 possible bytes
> Recording extents for root 9
> processed 16384 of 16384 possible bytes
> Recording extents for root 11221
> processed 16384 of 255983616 possible bytes
> Recording extents for root 11222
> processed 49479680 of 49479680 possible bytes
> Recording extents for root 11223
> processed 1635319808 of 1635549184 possible bytes
> Recording extents for root 11224
> processed 75792384 of 75792384 possible bytes
> Recording extents for root 159785
> processed 108855296 of 108855296 possible bytes
> Recording extents for root 159787
> processed 49152 of 49479680 possible bytes
> Recording extents for root 160494
> processed 1179648 of 109035520 possible bytes
> Recording extents for root 160496
> processed 49152 of 49479680 possible bytes
> Recording extents for root 161197
> processed 147456 of 109019136 possible bytes
> Recording extents for root 161199
> processed 49152 of 49479680 possible bytes
> Recording extents for root 162628
> processed 49152 of 49479680 possible bytes
> Recording extents for root 162632
> processed 2129920 of 109314048 possible bytes
> Recording extents for root 162645
> processed 49152 of 75792384 possible bytes
> Recording extents for root 163298
> processed 49152 of 49479680 possible bytes
> Recording extents for root 163302
> processed 147456 of 109314048 possible bytes
> Recording extents for root 163303
> processed 81920 of 75792384 possible bytes
> Recording extents for root 163316
> processed 49152 of 109314048 possible bytes
> Recording extents for root 163318
> processed 16384 of 49479680 possible bytes
> Recording extents for root 163916
> processed 49152 of 49479680 possible bytes
> Recording extents for root 163920
> processed 81920 of 109314048 possible bytes
> Recording extents for root 163921
> processed 49152 of 75792384 possible bytes
> Recording extents for root 164620
> processed 49152 of 49479680 possible bytes
> Recording extents for root 164624
> processed 491520 of 109445120 possible bytes
> Recording extents for root 164633
> processed 49152 of 75792384 possible bytes
> Recording extents for root 165098
> processed 212992 of 109445120 possible bytes
> Recording extents for root 165100
> processed 16384 of 49479680 possible bytes
> Recording extents for root 165198
> processed 49152 of 109445120 possible bytes
> Recording extents for root 165200
> processed 16384 of 49479680 possible bytes
> Recording extents for root 165294
> processed 16384 of 49479680 possible bytes
> Recording extents for root 165298
> processed 81920 of 109445120 possible bytes
> Recording extents for root 165299
> processed 16384 of 75792384 possible bytes
> Recording extents for root 18446744073709551607
> processed 16384 of 16384 possible bytes
> doing block accounting
> doing close???

Ok must be in the block accounting stuff which has 0 prints, fixed
that up.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-01 11:28                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-05-01 15:22                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-05-01 23:09                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-01 15:22 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 01, 2022 at 07:28:10AM -0400, Josef Bacik wrote:
> > processed 81920 of 109445120 possible bytes
> > Recording extents for root 165299
> > processed 16384 of 75792384 possible bytes
> > Recording extents for root 18446744073709551607
> > processed 16384 of 16384 possible bytes
> > doing block accounting
> > doing close???
> 
> Ok must be in the block accounting stuff which has 0 prints, fixed
> that up.  Thanks,

Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
ERROR: update block group failed
FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

doing close???
ERROR: commit_root already set when starting transaction
extent buffer leak: start

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-01 15:22                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-05-01 23:09                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-05-02  1:25                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-01 23:09 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 1, 2022 at 11:22 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 01, 2022 at 07:28:10AM -0400, Josef Bacik wrote:
> > > processed 81920 of 109445120 possible bytes
> > > Recording extents for root 165299
> > > processed 16384 of 75792384 possible bytes
> > > Recording extents for root 18446744073709551607
> > > processed 16384 of 16384 possible bytes
> > > doing block accounting
> > > doing close???
> >
> > Ok must be in the block accounting stuff which has 0 prints, fixed
> > that up.  Thanks,
>
> Recording extents for root 18446744073709551607
> processed 16384 of 16384 possible bytes
> doing block accounting
> ERROR: update block group failed
> FIX BLOCK ACCOUNTING FAILED -1
> ERROR: The commit failed???? -1
>

Sorry was on airplanes, pushed some more debugging.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-01 23:09                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-05-02  1:25                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-05-02 16:41                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-02  1:25 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 01, 2022 at 07:09:19PM -0400, Josef Bacik wrote:
> Sorry was on airplanes, pushed some more debugging.  Thanks,

no worries

Recording extents for root 165294
processed 16384 of 49479680 possible bytes
Recording extents for root 165298
processed 81920 of 109445120 possible bytes
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
ERROR: update block group failed 15847955365888 1073217536 ret -1
FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

doing close???
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13576823767040 len 16384
Init extent tree failed
[Inferior 1 (process 23375) exited with code 0377]

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-02  1:25                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-05-02 16:41                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-05-02 17:34                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-02 16:41 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 1, 2022 at 9:25 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 01, 2022 at 07:09:19PM -0400, Josef Bacik wrote:
> > Sorry was on airplanes, pushed some more debugging.  Thanks,
>
> no worries
>
> Recording extents for root 165294
> processed 16384 of 49479680 possible bytes
> Recording extents for root 165298
> processed 81920 of 109445120 possible bytes
> Recording extents for root 165299
> processed 16384 of 75792384 possible bytes
> Recording extents for root 18446744073709551607
> processed 16384 of 16384 possible bytes
> doing block accounting
> ERROR: update block group failed 15847955365888 1073217536 ret -1
> FIX BLOCK ACCOUNTING FAILED -1
> ERROR: The commit failed???? -1
>

More debugging, this really shouldn't happen because we wouldn't be
able to map the bytenr.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-02 16:41                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-05-02 17:34                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-05-02 19:07                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-02 17:34 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, May 02, 2022 at 12:41:11PM -0400, Josef Bacik wrote:
> On Sun, May 1, 2022 at 9:25 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sun, May 01, 2022 at 07:09:19PM -0400, Josef Bacik wrote:
> > > Sorry was on airplanes, pushed some more debugging.  Thanks,
> >
> > no worries
> >
> > Recording extents for root 165294
> > processed 16384 of 49479680 possible bytes
> > Recording extents for root 165298
> > processed 81920 of 109445120 possible bytes
> > Recording extents for root 165299
> > processed 16384 of 75792384 possible bytes
> > Recording extents for root 18446744073709551607
> > processed 16384 of 16384 possible bytes
> > doing block accounting
> > ERROR: update block group failed 15847955365888 1073217536 ret -1
> > FIX BLOCK ACCOUNTING FAILED -1
> > ERROR: The commit failed???? -1
> >
> 
> More debugging, this really shouldn't happen because we wouldn't be
> able to map the bytenr.  Thanks,

Sure thing:
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
couldn't find a block group at bytenr 15847955365888 total left 1073217536
ERROR: update block group failed 15847955365888 1073217536 ret -1
FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

doing close???
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13576823767040 len 16384
Init extent tree failed



-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-02 17:34                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-05-02 19:07                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-05-02 20:08                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-02 19:07 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 2, 2022 at 1:35 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 02, 2022 at 12:41:11PM -0400, Josef Bacik wrote:
> > On Sun, May 1, 2022 at 9:25 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sun, May 01, 2022 at 07:09:19PM -0400, Josef Bacik wrote:
> > > > Sorry was on airplanes, pushed some more debugging.  Thanks,
> > >
> > > no worries
> > >
> > > Recording extents for root 165294
> > > processed 16384 of 49479680 possible bytes
> > > Recording extents for root 165298
> > > processed 81920 of 109445120 possible bytes
> > > Recording extents for root 165299
> > > processed 16384 of 75792384 possible bytes
> > > Recording extents for root 18446744073709551607
> > > processed 16384 of 16384 possible bytes
> > > doing block accounting
> > > ERROR: update block group failed 15847955365888 1073217536 ret -1
> > > FIX BLOCK ACCOUNTING FAILED -1
> > > ERROR: The commit failed???? -1
> > >
> >
> > More debugging, this really shouldn't happen because we wouldn't be
> > able to map the bytenr.  Thanks,
>
> Sure thing:
> Recording extents for root 165299
> processed 16384 of 75792384 possible bytes
> Recording extents for root 18446744073709551607
> processed 16384 of 16384 possible bytes
> doing block accounting
> couldn't find a block group at bytenr 15847955365888 total left 1073217536
> ERROR: update block group failed 15847955365888 1073217536 ret -1
> FIX BLOCK ACCOUNTING FAILED -1
> ERROR: The commit failed???? -1
>
> doing close???
> ERROR: commit_root already set when starting transaction
> extent buffer leak: start 13576823767040 len 16384
> Init extent tree failed
>

Wtf, that's definitely in the range of block groups that I've seen
printed.  I added some more stuff, hoping the tree search is just
broken.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-02 19:07                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-05-02 20:08                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-05-02 21:03                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-02 20:08 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, May 02, 2022 at 03:07:09PM -0400, Josef Bacik wrote:
> > Sure thing:
> > Recording extents for root 165299
> > processed 16384 of 75792384 possible bytes
> > Recording extents for root 18446744073709551607
> > processed 16384 of 16384 possible bytes
> > doing block accounting
> > couldn't find a block group at bytenr 15847955365888 total left 1073217536
> > ERROR: update block group failed 15847955365888 1073217536 ret -1
> > FIX BLOCK ACCOUNTING FAILED -1
> > ERROR: The commit failed???? -1
> >
> > doing close???
> > ERROR: commit_root already set when starting transaction
> > extent buffer leak: start 13576823767040 len 16384
> > Init extent tree failed
> >
> 
> Wtf, that's definitely in the range of block groups that I've seen
> printed.  I added some more stuff, hoping the tree search is just
> broken.  Thanks,

Recording extents for root 163303
processed 81920 of 75792384 possible bytes
inserting block group 15731991248896
inserting block group 15733064990720
inserting block group 15734138732544
inserting block group 15735212474368
inserting block group 15736286216192
inserting block group 15737359958016
inserting block group 15738433699840
inserting block group 15739507441664
inserting block group 15740581183488
inserting block group 15741654925312
inserting block group 15742728667136
inserting block group 15743802408960
inserting block group 15744876150784
inserting block group 15745949892608
inserting block group 15747023634432
inserting block group 15748097376256
inserting block group 15749171118080
inserting block group 15750244859904
inserting block group 15751318601728
inserting block group 15752392343552
inserting block group 15753466085376
inserting block group 15754539827200
inserting block group 15755613569024
inserting block group 15756687310848
inserting block group 15757761052672
inserting block group 15758834794496
inserting block group 15759908536320
inserting block group 15760982278144
inserting block group 15762056019968
inserting block group 15763129761792
inserting block group 15764203503616
inserting block group 15765277245440
inserting block group 15766350987264
inserting block group 15767424729088
inserting block group 15768498470912
inserting block group 15769572212736
inserting block group 15770645954560
inserting block group 15771719696384
inserting block group 15778162147328
inserting block group 15779235889152
inserting block group 15780309630976
inserting block group 15781383372800
inserting block group 15782457114624
inserting block group 15783530856448
inserting block group 15784604598272
inserting block group 15785678340096
inserting block group 15786752081920
inserting block group 15787825823744
inserting block group 15788899565568
inserting block group 15789973307392
inserting block group 15791047049216
inserting block group 15792120791040
inserting block group 15793194532864
inserting block group 15794268274688
inserting block group 15795342016512
inserting block group 15796415758336
inserting block group 15797489500160
inserting block group 15798563241984
inserting block group 15799636983808
inserting block group 15800710725632
inserting block group 15801784467456
inserting block group 15802858209280
inserting block group 15803931951104
inserting block group 15809300660224
inserting block group 15810374402048
inserting block group 15811448143872
inserting block group 15812521885696
inserting block group 15813595627520
inserting block group 15814669369344
inserting block group 15815743111168
inserting block group 15816816852992
inserting block group 15817890594816
inserting block group 15818964336640
inserting block group 15820038078464
inserting block group 15821111820288
inserting block group 15822185562112
cache 15738433699840 15739507441664 in range no
cache 15739507441664 15740581183488 in range no
cache 15740581183488 15741654925312 in range no
cache 15741654925312 15742728667136 in range no
cache 15742728667136 15743802408960 in range no
cache 15743802408960 15744876150784 in range no
cache 15744876150784 15745949892608 in range no
cache 15745949892608 15747023634432 in range no
cache 15747023634432 15748097376256 in range no
cache 15748097376256 15749171118080 in range no
cache 15749171118080 15750244859904 in range no
cache 15750244859904 15751318601728 in range no
cache 15751318601728 15752392343552 in range no
cache 15752392343552 15753466085376 in range no
cache 15753466085376 15754539827200 in range no
cache 15754539827200 15755613569024 in range no
cache 15755613569024 15756687310848 in range no
cache 15756687310848 15757761052672 in range no
cache 15757761052672 15758834794496 in range no
cache 15758834794496 15759908536320 in range no
cache 15759908536320 15760982278144 in range no
cache 15760982278144 15762056019968 in range no
cache 15762056019968 15763129761792 in range no
cache 15763129761792 15764203503616 in range no
cache 15764203503616 15765277245440 in range no
cache 15765277245440 15766350987264 in range no
cache 15766350987264 15767424729088 in range no
cache 15767424729088 15768498470912 in range no
cache 15768498470912 15769572212736 in range no
cache 15769572212736 15770645954560 in range no
cache 15770645954560 15771719696384 in range no
cache 15771719696384 15772793438208 in range no
cache 15778162147328 15779235889152 in range no
cache 15779235889152 15780309630976 in range no
cache 15780309630976 15781383372800 in range no
cache 15781383372800 15782457114624 in range no
cache 15782457114624 15783530856448 in range no
cache 15783530856448 15784604598272 in range no
cache 15784604598272 15785678340096 in range no
cache 15785678340096 15786752081920 in range no
cache 15786752081920 15787825823744 in range no
cache 15787825823744 15788899565568 in range no
cache 15788899565568 15789973307392 in range no
cache 15789973307392 15791047049216 in range no
cache 15791047049216 15792120791040 in range no
cache 15792120791040 15793194532864 in range no
cache 15793194532864 15794268274688 in range no
cache 15794268274688 15795342016512 in range no
cache 15795342016512 15796415758336 in range no
cache 15796415758336 15797489500160 in range no
cache 15797489500160 15798563241984 in range no
cache 15798563241984 15799636983808 in range no
cache 15799636983808 15800710725632 in range no
cache 15800710725632 15801784467456 in range no
cache 15801784467456 15802858209280 in range no
cache 15802858209280 15803931951104 in range no
cache 15803931951104 15805005692928 in range no
cache 15809300660224 15810374402048 in range no
cache 15810374402048 15811448143872 in range no
cache 15811448143872 15812521885696 in range no
cache 15812521885696 15813595627520 in range no
cache 15813595627520 15814669369344 in range no
cache 15814669369344 15815743111168 in range no
cache 15815743111168 15816816852992 in range no
cache 15816816852992 15817890594816 in range no
cache 15817890594816 15818964336640 in range no
cache 15818964336640 15820038078464 in range no
cache 15820038078464 15821111820288 in range no
cache 15821111820288 15822185562112 in range no
cache 15822185562112 15823259303936 in range no
cache 15823259303936 15824333045760 in range no
cache 15824333045760 15825406787584 in range no
cache 15825406787584 15826480529408 in range no
cache 15826480529408 15827554271232 in range no
cache 15827554271232 15828628013056 in range no
cache 15828628013056 15829701754880 in range no
cache 15829701754880 15830775496704 in range no
cache 15830775496704 15831849238528 in range no
cache 15831849238528 15832922980352 in range no
cache 15832922980352 15833996722176 in range no
cache 15833996722176 15835070464000 in range no
cache 15835070464000 15836144205824 in range no
cache 15836144205824 15837217947648 in range no
cache 15837217947648 15838291689472 in range no
cache 15838291689472 15839365431296 in range no
cache 15839365431296 15840439173120 in range no
cache 15840439173120 15841512914944 in range no
cache 15842586656768 15843660398592 in range no
ERROR: update block group failed 15847955365888 1073217536 ret -1
FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

doing close???
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13576823816192 len 16384
Init extent tree failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-02 20:08                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-05-02 21:03                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-05-02 21:49                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-02 21:03 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 2, 2022 at 4:08 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 02, 2022 at 03:07:09PM -0400, Josef Bacik wrote:
> > > Sure thing:
> > > Recording extents for root 165299
> > > processed 16384 of 75792384 possible bytes
> > > Recording extents for root 18446744073709551607
> > > processed 16384 of 16384 possible bytes
> > > doing block accounting
> > > couldn't find a block group at bytenr 15847955365888 total left 1073217536
> > > ERROR: update block group failed 15847955365888 1073217536 ret -1
> > > FIX BLOCK ACCOUNTING FAILED -1
> > > ERROR: The commit failed???? -1
> > >
> > > doing close???
> > > ERROR: commit_root already set when starting transaction
> > > extent buffer leak: start 13576823767040 len 16384
> > > Init extent tree failed
> > >
> >
> > Wtf, that's definitely in the range of block groups that I've seen
> > printed.  I added some more stuff, hoping the tree search is just
> > broken.  Thanks,
>

Ok I've fixed it to yell about what file has this weird extent so you
can delete it and we can carry on.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-02 21:03                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-05-02 21:49                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-05-02 23:16                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-02 21:49 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, May 02, 2022 at 05:03:40PM -0400, Josef Bacik wrote:
> Ok I've fixed it to yell about what file has this weird extent so you
> can delete it and we can carry on.  Thanks,

That worked.  How do I delete this one?

doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819130,108,0 on root 11223
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0027_2020.1_0602_1208.xz
cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON `1` triggered, value 1
./btrfs(+0x8b9fc)[0x5575fbec99fc]
./btrfs(+0x8c0df)[0x5575fbeca0df]
./btrfs(+0x8c22d)[0x5575fbeca22d]
./btrfs(+0x8c22d)[0x5575fbeca22d]
./btrfs(+0x8c49b)[0x5575fbeca49b]
./btrfs(btrfs_init_extent_tree+0xe03)[0x5575fbecb37b]
./btrfs(+0x83b7d)[0x5575fbec1b7d]
./btrfs(handle_command_group+0x49)[0x5575fbe5617b]
./btrfs(main+0x94)[0x5575fbe56275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fa2455a77fd]
./btrfs(_start+0x2a)[0x5575fbe55e1a]
Aborted


Thanks
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-02 21:49                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-05-02 23:16                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-05-02 23:41                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-02 23:16 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 2, 2022 at 2:49 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 02, 2022 at 05:03:40PM -0400, Josef Bacik wrote:
> > Ok I've fixed it to yell about what file has this weird extent so you
> > can delete it and we can carry on.  Thanks,
>
> That worked.  How do I delete this one?
>
> doing roots
> Recording extents for root 4
> processed 1032192 of 1064960 possible bytes
> Recording extents for root 5
> processed 10960896 of 10977280 possible bytes
> Recording extents for root 7
> processed 16384 of 16545742848 possible bytes
> Recording extents for root 9
> processed 16384 of 16384 possible bytes
> Recording extents for root 11221
> processed 16384 of 255983616 possible bytes
> Recording extents for root 11222
> processed 49479680 of 49479680 possible bytes
> Recording extents for root 11223
> processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819130,108,0 on root 11223

btrfs-corrupt-block -d "1819130,108,0" -r 11223 <device>

Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-02 23:16                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-05-02 23:41                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-05-03  1:06                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-02 23:41 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, May 02, 2022 at 04:16:14PM -0700, Josef Bacik wrote:
> On Mon, May 2, 2022 at 2:49 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Mon, May 02, 2022 at 05:03:40PM -0400, Josef Bacik wrote:
> > > Ok I've fixed it to yell about what file has this weird extent so you
> > > can delete it and we can carry on.  Thanks,
> >
> > That worked.  How do I delete this one?
> >
> > doing roots
> > Recording extents for root 4
> > processed 1032192 of 1064960 possible bytes
> > Recording extents for root 5
> > processed 10960896 of 10977280 possible bytes
> > Recording extents for root 7
> > processed 16384 of 16545742848 possible bytes
> > Recording extents for root 9
> > processed 16384 of 16384 possible bytes
> > Recording extents for root 11221
> > processed 16384 of 255983616 possible bytes
> > Recording extents for root 11222
> > processed 49479680 of 49479680 possible bytes
> > Recording extents for root 11223
> > processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819130,108,0 on root 11223
> 
> btrfs-corrupt-block -d "1819130,108,0" -r 11223 <device>

Whoops, it was right there in the text, sorry

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819130,108,0" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x564360382600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x564360382600


Didn't help?
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819131,108,0 on root 11223
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0026_2020.1_0602_1208.xz
cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON `1` triggered, value 1

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-02 23:41                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-05-03  1:06                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-05-03  1:26                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-03  1:06 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 2, 2022 at 7:41 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 02, 2022 at 04:16:14PM -0700, Josef Bacik wrote:
> > On Mon, May 2, 2022 at 2:49 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Mon, May 02, 2022 at 05:03:40PM -0400, Josef Bacik wrote:
> > > > Ok I've fixed it to yell about what file has this weird extent so you
> > > > can delete it and we can carry on.  Thanks,
> > >
> > > That worked.  How do I delete this one?
> > >
> > > doing roots
> > > Recording extents for root 4
> > > processed 1032192 of 1064960 possible bytes
> > > Recording extents for root 5
> > > processed 10960896 of 10977280 possible bytes
> > > Recording extents for root 7
> > > processed 16384 of 16545742848 possible bytes
> > > Recording extents for root 9
> > > processed 16384 of 16384 possible bytes
> > > Recording extents for root 11221
> > > processed 16384 of 255983616 possible bytes
> > > Recording extents for root 11222
> > > processed 49479680 of 49479680 possible bytes
> > > Recording extents for root 11223
> > > processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819130,108,0 on root 11223
> >
> > btrfs-corrupt-block -d "1819130,108,0" -r 11223 <device>
>
> Whoops, it was right there in the text, sorry
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819130,108,0" -r 11223 /dev/mapper/dshelf1
> FS_INFO IS 0x564360382600
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x564360382600
>
>
> Didn't help?
> doing roots
> Recording extents for root 4
> processed 1032192 of 1064960 possible bytes
> Recording extents for root 5
> processed 10960896 of 10977280 possible bytes
> Recording extents for root 7
> processed 16384 of 16545742848 possible bytes
> Recording extents for root 9
> processed 16384 of 16384 possible bytes
> Recording extents for root 11221
> processed 16384 of 255983616 possible bytes
> Recording extents for root 11222
> processed 49479680 of 49479680 possible bytes
> Recording extents for root 11223
> processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819131,108,0 on root 11223

It's a different inode number, 1819131 instead of 1819130.  This is
going to be the frustrating delete shit until it works thing again.
Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-03  1:06                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-05-03  1:26                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-05-03  2:38                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-03  1:26 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, May 02, 2022 at 09:06:50PM -0400, Josef Bacik wrote:
> It's a different inode number, 1819131 instead of 1819130.  This is
> going to be the frustrating delete shit until it works thing again.

Indeed, I missed that.

Now I'm stuck here:
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,0 on root 11223
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xz


gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x55ca68c6d600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55ca68c6d600
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
couldn't find root 11223

Mmmh, that's not great looking....

doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,0 on root 11223
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xz


gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x56267a388600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x56267a388600
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
couldn't find root 11223

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-03  1:26                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-05-03  2:38                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-05-03  4:02                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-03  2:38 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 2, 2022 at 9:26 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 02, 2022 at 09:06:50PM -0400, Josef Bacik wrote:
> > It's a different inode number, 1819131 instead of 1819130.  This is
> > going to be the frustrating delete shit until it works thing again.
>
> Indeed, I missed that.
>
> Now I'm stuck here:
> Recording extents for root 11223
> processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,0 on root 11223
> inode ref info failed???
> elem_cnt 1 elem_missed 0 ret 0
> Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xz
>
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1
> FS_INFO IS 0x55ca68c6d600
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x55ca68c6d600
> parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
> parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
> couldn't find root 11223
>
> Mmmh, that's not great looking....
>

Ugh IDK why that happens every once and a while.  I pushed a fix for
btrfs-corrupt-block, it should work now.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-03  2:38                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-05-03  4:02                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-05-03  4:13                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-03  4:02 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, May 02, 2022 at 10:38:03PM -0400, Josef Bacik wrote:
> Ugh IDK why that happens every once and a while.  I pushed a fix for
> btrfs-corrupt-block, it should work now.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x562d920a7600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x562d920a7600
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
FS_INFO IS 0x5649e173dbc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5649e173dbc0
Walking all our trees and pinning down the currently accessible blocks
(..,)

doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,134217728 on root 11223
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xz
cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON `1` triggered, value 1

Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819135,108,0 on root 11223
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/ise_0007_2020.1_0602_1208.xz
cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON `1` triggered, value 1

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819135,108,0" -r 11223 /dev/mapper/dshelf1 

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819137,108,0" -r 11223 /dev/mapper/dshelf1 

Ok, now we're down to another one:
(...)
        item 143 key (13577065349120 METADATA_ITEM 0) itemoff 11801 itemsize 33
                refs 1 gen 1451459 flags TREE_BLOCK
                tree block skinny level 0
                tree block backref root FS_TREE
        item 144 key (13577069330432 METADATA_ITEM 0) itemoff 11768 itemsize 33
                refs 1 gen 1538467 flags TREE_BLOCK|FULL_BACKREF
                tree block skinny level 0
                shared block backref parent 15645094871040
        item 145 key (13577073082368 METADATA_ITEM 0) itemoff 11735 itemsize 33
                refs 1 gen 1451996 flags TREE_BLOCK
                tree block skinny level 0
                tree block backref root FS_TREE
        item 146 key (13577075130368 METADATA_ITEM 0) itemoff 11702 itemsize 33
                refs 1 gen 1452003 flags TREE_BLOCK
                tree block skinny level 0
                tree block backref root FS_TREE
        item 147 key (13577080455168 METADATA_ITEM 0) itemoff 11669 itemsize 33
                refs 1 gen 1452018 flags TREE_BLOCK
                tree block skinny level 0
                tree block backref root FS_TREE
        item 148 key (13577081552896 METADATA_ITEM 0) itemoff 11636 itemsize 33
                refs 1 gen 1452010 flags TREE_BLOCK
                tree block skinny level 0
                tree block backref root FS_TREE
        item 149 key (13577082077184 METADATA_ITEM 0) itemoff 11603 itemsize 33
                refs 1 gen 1538321 flags TREE_BLOCK|FULL_BACKREF
                tree block skinny level 0
                shared block backref parent 15645094871040
        item 150 key (13577082208256 METADATA_ITEM 0) itemoff 11570 itemsize 33
                refs 1 gen 1538467 flags TREE_BLOCK|FULL_BACKREF
                tree block skinny level 0
                shared block backref parent 15645094871040
        item 151 key (13577082552320 METADATA_ITEM 0) itemoff 11537 itemsize 33
                refs 1 gen 1452024 flags TREE_BLOCK
                tree block skinny level 0
                tree block backref root FS_TREE
        item 152 key (13577085927424 METADATA_ITEM 0) itemoff 11504 itemsize 33
                refs 1 gen 1452031 flags TREE_BLOCK
                tree block skinny level 0
                tree block backref root FS_TREE
        item 153 key (13577088073728 METADATA_ITEM 0) itemoff 11471 itemsize 33
                refs 1 gen 1443442 flags TREE_BLOCK
                tree block skinny level 0
                tree block backref root DEV_TREE
        item 154 key (13577088696320 METADATA_ITEM 0) itemoff 11438 itemsize 33
                refs 1 gen 1452038 flags TREE_BLOCK
                tree block skinny level 0
                tree block backref root FS_TREE
        item 155 key (13577090547712 METADATA_ITEM 0) itemoff 11405 itemsize 33
                refs 1 gen 1590216 flags TREE_BLOCK
                tree block skinny level 0
                tree block backref root 11223
        item 156 key (13577090596864 METADATA_ITEM 0) itemoff 11372 itemsize 33
                refs 1 gen 1590216 flags TREE_BLOCK
                tree block skinny level 0
                tree block backref root 11223
        item 157 key (13577091399680 METADATA_ITEM 0) itemoff 11339 itemsize 33
                refs 1 gen 1538745 flags TREE_BLOCK|FULL_BACKREF
                tree block skinny level 0
                shared block backref parent 15645094871040
        item 158 key (13577093103616 METADATA_ITEM 0) itemoff 11306 itemsize 33
                refs 1 gen 1452046 flags TREE_BLOCK
                tree block skinny level 0
                tree block backref root FS_TREE
        item 159 key (13577095495680 METADATA_ITEM 0) itemoff 11273 itemsize 33
                refs 1 gen 1452053 flags TREE_BLOCK
                tree block skinny level 0
                tree block backref root FS_TREE
        item 160 key (13577095544832 METADATA_ITEM 0) itemoff 11240 itemsize 33
                refs 1 gen 1538650 flags TREE_BLOCK|FULL_BACKREF
                tree block skinny level 0
                shared block backref parent 15645094871040
        item 161 key (13577099837440 METADATA_ITEM 0) itemoff 11207 itemsize 33
                refs 1 gen 1452063 flags TREE_BLOCK
                tree block skinny level 0
                tree block backref root FS_TREE
failed to find block number 13576823652352
kernel-shared/extent-tree.c:1432: btrfs_set_block_flags: BUG_ON `1` triggered, value 1
./btrfs(+0x297a4)[0x56509942c7a4]
./btrfs(+0x297f4)[0x56509942c7f4]
./btrfs(btrfs_set_block_flags+0x1fb)[0x56509942f623]
./btrfs(+0x8be0e)[0x56509948ee0e]
./btrfs(+0x8c22d)[0x56509948f22d]
./btrfs(+0x8c22d)[0x56509948f22d]
./btrfs(+0x8c49b)[0x56509948f49b]
./btrfs(btrfs_init_extent_tree+0xe03)[0x56509949037b]
./btrfs(+0x83b7d)[0x565099486b7d]
./btrfs(handle_command_group+0x49)[0x56509941b17b]
./btrfs(main+0x94)[0x56509941b275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7feced8517fd]
./btrfs(_start+0x2a)[0x56509941ae1a]
Aborted


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-03  4:02                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-05-03  4:13                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-05-03  4:55                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-03  4:13 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, May 3, 2022 at 12:02 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 02, 2022 at 10:38:03PM -0400, Josef Bacik wrote:
> > Ugh IDK why that happens every once and a while.  I pushed a fix for
> > btrfs-corrupt-block, it should work now.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1
> FS_INFO IS 0x562d920a7600
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x562d920a7600
> parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
> parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
> parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
> FS_INFO IS 0x5649e173dbc0
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x5649e173dbc0
> Walking all our trees and pinning down the currently accessible blocks
> (..,)
>
> doing roots
> Recording extents for root 4
> processed 1032192 of 1064960 possible bytes
> Recording extents for root 5
> processed 10960896 of 10977280 possible bytes
> Recording extents for root 7
> processed 16384 of 16545742848 possible bytes
> Recording extents for root 9
> processed 16384 of 16384 possible bytes
> Recording extents for root 11221
> processed 16384 of 255983616 possible bytes
> Recording extents for root 11222
> processed 49479680 of 49479680 possible bytes
> Ignoring transid failure
> Recording extents for root 11223
> processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,134217728 on root 11223
> inode ref info failed???
> elem_cnt 1 elem_missed 0 ret 0
> Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xz
> cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON `1` triggered, value 1
>
> Ignoring transid failure
> Recording extents for root 11223
> processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819135,108,0 on root 11223
> inode ref info failed???
> elem_cnt 1 elem_missed 0 ret 0
> Xilinx_Unified_2020.1_0602_1208/payload/ise_0007_2020.1_0602_1208.xz
> cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON `1` triggered, value 1
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819135,108,0" -r 11223 /dev/mapper/dshelf1
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819137,108,0" -r 11223 /dev/mapper/dshelf1
>
> Ok, now we're down to another one:

Ok I fixed the debugging to not be so noisy so I can see what's going
on, lets give that a try,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-03  4:13                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-05-03  4:55                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-05-03 16:00                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-03  4:55 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, May 03, 2022 at 12:13:16AM -0400, Josef Bacik wrote:
> Ok I fixed the debugging to not be so noisy so I can see what's going
> on, lets give that a try,

(...)
inserting block group 13570381185024
inserting block group 13571454926848
inserting block group 13572528668672
inserting block group 13573602410496
inserting block group 13574676152320
inserting block group 13575749894144
inserting block group 13576823635968
inserting block group 13577897377792
inserting block group 13585413570560
(...)
inserting block group 15768498470912
inserting block group 15769572212736
inserting block group 15770645954560
inserting block group 15771719696384
inserting block group 15778162147328
inserting block group 15779235889152
inserting block group 15780309630976
inserting block group 15781383372800
inserting block group 15782457114624
inserting block group 15783530856448
inserting block group 15784604598272
inserting block group 15785678340096
inserting block group 15786752081920
inserting block group 15787825823744
inserting block group 15788899565568
inserting block group 15789973307392
inserting block group 15791047049216
inserting block group 15792120791040
inserting block group 15793194532864
inserting block group 15794268274688
inserting block group 15795342016512
inserting block group 15796415758336
inserting block group 15797489500160
inserting block group 15798563241984
inserting block group 15799636983808
inserting block group 15800710725632
inserting block group 15801784467456
inserting block group 15802858209280
inserting block group 15803931951104
inserting block group 15809300660224
inserting block group 15810374402048
inserting block group 15811448143872
inserting block group 15812521885696
inserting block group 15813595627520
inserting block group 15814669369344
inserting block group 15815743111168
inserting block group 15816816852992
inserting block group 15817890594816
inserting block group 15818964336640
inserting block group 15820038078464
inserting block group 15821111820288
inserting block group 15822185562112
inserting block group 15823259303936
inserting block group 15824333045760
inserting block group 15825406787584
inserting block group 15826480529408
inserting block group 15827554271232
inserting block group 15828628013056
inserting block group 15829701754880
inserting block group 15830775496704
inserting block group 15831849238528
inserting block group 15832922980352
inserting block group 15833996722176
inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesIgnoring transid failure
failed to find block number 13576823652352
kernel-shared/extent-tree.c:1432: btrfs_set_block_flags: BUG_ON `1` triggered, value 1

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-03  4:55                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-05-03 16:00                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-05-03 17:24                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-03 16:00 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, May 3, 2022 at 12:55 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, May 03, 2022 at 12:13:16AM -0400, Josef Bacik wrote:
> > Ok I fixed the debugging to not be so noisy so I can see what's going
> > on, lets give that a try,
>
> (...)
> inserting block group 13570381185024
> inserting block group 13571454926848
> inserting block group 13572528668672
> inserting block group 13573602410496
> inserting block group 13574676152320
> inserting block group 13575749894144
> inserting block group 13576823635968
> inserting block group 13577897377792
> inserting block group 13585413570560
> (...)
> inserting block group 15768498470912
> inserting block group 15769572212736
> inserting block group 15770645954560
> inserting block group 15771719696384
> inserting block group 15778162147328
> inserting block group 15779235889152
> inserting block group 15780309630976
> inserting block group 15781383372800
> inserting block group 15782457114624
> inserting block group 15783530856448
> inserting block group 15784604598272
> inserting block group 15785678340096
> inserting block group 15786752081920
> inserting block group 15787825823744
> inserting block group 15788899565568
> inserting block group 15789973307392
> inserting block group 15791047049216
> inserting block group 15792120791040
> inserting block group 15793194532864
> inserting block group 15794268274688
> inserting block group 15795342016512
> inserting block group 15796415758336
> inserting block group 15797489500160
> inserting block group 15798563241984
> inserting block group 15799636983808
> inserting block group 15800710725632
> inserting block group 15801784467456
> inserting block group 15802858209280
> inserting block group 15803931951104
> inserting block group 15809300660224
> inserting block group 15810374402048
> inserting block group 15811448143872
> inserting block group 15812521885696
> inserting block group 15813595627520
> inserting block group 15814669369344
> inserting block group 15815743111168
> inserting block group 15816816852992
> inserting block group 15817890594816
> inserting block group 15818964336640
> inserting block group 15820038078464
> inserting block group 15821111820288
> inserting block group 15822185562112
> inserting block group 15823259303936
> inserting block group 15824333045760
> inserting block group 15825406787584
> inserting block group 15826480529408
> inserting block group 15827554271232
> inserting block group 15828628013056
> inserting block group 15829701754880
> inserting block group 15830775496704
> inserting block group 15831849238528
> inserting block group 15832922980352
> inserting block group 15833996722176
> inserting block group 15835070464000
> inserting block group 15836144205824
> inserting block group 15837217947648
> inserting block group 15838291689472
> inserting block group 15839365431296
> inserting block group 15840439173120
> inserting block group 15842586656768
> processed 1556480 of 0 possible bytes
> processed 1474560 of 0 possible bytes
> doing roots
> Recording extents for root 4
> processed 1032192 of 1064960 possible bytes
> Recording extents for root 5
> processed 10960896 of 10977280 possible bytes
> Recording extents for root 7
> processed 16384 of 16545742848 possible bytes
> Recording extents for root 9
> processed 16384 of 16384 possible bytes
> Recording extents for root 11221
> processed 16384 of 255983616 possible bytes
> Recording extents for root 11222
> processed 49479680 of 49479680 possible bytes
> Ignoring transid failure
> Recording extents for root 11223
> processed 1619902464 of 1635549184 possible bytesIgnoring transid failure
> failed to find block number 13576823652352
> kernel-shared/extent-tree.c:1432: btrfs_set_block_flags: BUG_ON `1` triggered, value 1
>

Fucking hell, do the tree-recover again and then the init-extent-tree.
Once I'm done with this conference I'll work out why this keeps
happening.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-03 16:00                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-05-03 17:24                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-05-05 15:08                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-03 17:24 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, May 03, 2022 at 12:00:56PM -0400, Josef Bacik wrote:
> > Recording extents for root 11223
> > processed 1619902464 of 1635549184 possible bytesIgnoring transid failure
> > failed to find block number 13576823652352
> > kernel-shared/extent-tree.c:1432: btrfs_set_block_flags: BUG_ON `1` triggered, value 1
> 
> Fucking hell, do the tree-recover again and then the init-extent-tree.
> Once I'm done with this conference I'll work out why this keeps
> happening.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x561ce6c0cbc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x561ce6c0cbc0
Checking root 2 bytenr 15645018603520
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 15645018521600
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736
Checking root 11222 bytenr 15645261905920
Checking root 11223 bytenr 13576823635968
Repairing root 11223 bad_blocks 1 update 1
Repairing root 163303 bad_blocks 1 update 0
updating slot 11 in block 15645018505216
updating slot 11 in block 15645018505216
updating slot 11 in block 15645018505216
updating slot 11 in block 15645018505216
updating slot 11 in block 15645018505216
updating slot 11 in block 15645018505216
updating slot 11 in block 15645018505216
deleting slot 11 in block 15645018505216
Checking root 163316 bytenr 6781245931520
Checking root 163318 bytenr 15645980491776
Checking root 163916 bytenr 11822437826560
Checking root 163920 bytenr 11970640084992
Checking root 163921 bytenr 11971073802240
Checking root 164620 bytenr 15645434036224
Checking root 164624 bytenr 15645018161152
Repairing root 164624 bad_blocks 1 update 1
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
deleting slot 24 in block 15645502210048
Checking root 164633 bytenr 15645526884352
Checking root 165098 bytenr 11970640101376
Repairing root 165098 bad_blocks 1 update 0
updating slot 425 in block 11821926875136
Checking root 165100 bytenr 11970733621248
Checking root 165198 bytenr 12511369756672
Repairing root 165198 bad_blocks 1 update 1
updating slot 425 in block 11822225473536
Checking root 165200 bytenr 12511677972480
Checking root 165294 bytenr 13576901328896
Checking root 165298 bytenr 13576823635968
Repairing root 165298 bad_blocks 0 update 1
Checking root 165299 bytenr 13577191505920
Checking root 18446744073709551607 bytenr 29540352
Tree recovery finished, you can run check now



gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x55ec18f10bc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55ec18f10bc0
Checking root 2 bytenr 15645018603520
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 15645018521600
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736
Checking root 11222 bytenr 15645261905920
Checking root 11223 bytenr 13576823635968
Checking root 11224 bytenr 13577126182912
Checking root 159785 bytenr 6781490577408
Checking root 159787 bytenr 15645908385792
Checking root 160494 bytenr 6781245882368
Checking root 160496 bytenr 11822309965824
Checking root 161197 bytenr 6781245865984
Checking root 161199 bytenr 13576850833408
Checking root 162628 bytenr 15645764812800
Checking root 162632 bytenr 6781245898752
Checking root 162645 bytenr 5809981095936
Checking root 163298 bytenr 15645124263936
Checking root 163302 bytenr 6781245915136
Checking root 163303 bytenr 15645018505216
Checking root 163316 bytenr 6781245931520
Checking root 163318 bytenr 15645980491776
Checking root 163916 bytenr 11822437826560
Checking root 163920 bytenr 11970640084992
Checking root 163921 bytenr 11971073802240
Checking root 164620 bytenr 15645434036224
Checking root 164624 bytenr 15645502210048
Checking root 164633 bytenr 15645526884352
Checking root 165098 bytenr 11970640101376
Checking root 165100 bytenr 11970733621248
Checking root 165198 bytenr 12511656394752
Checking root 165200 bytenr 12511677972480
Checking root 165294 bytenr 13576901328896
Checking root 165298 bytenr 13577133326336
Checking root 165299 bytenr 13577191505920
Checking root 18446744073709551607 bytenr 29540352
Tree recovery finished, you can run check now

Good, now running repair

gargamel:/var/local/src/btrfs-progs-josefbacik# .//btrfs check --repair /dev/mapper/dshelf1
enabling repair mode
WARNING:

        Do not use --repair unless you are advised to do so by a developer
        or an experienced user, and then only after having accepted that no
        fsck can successfully repair all types of filesystem corruption. Eg.
        some software or hardware bugs can fatally damage a volume.
        The operation will start in 10 seconds.
        Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
FS_INFO IS 0x5584836fafd0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5584836fafd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
Error: could not find extent items for root 11223
ERROR: failed to repair root items: No such file or directory


doing the longer one, will report back
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair --init-extent-tree /dev/mapper/dshelf1

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-03 17:24                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-05-05 15:08                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  2022-05-05 15:27                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-05 15:08 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, May 03, 2022 at 10:24:25AM -0700, Marc MERLIN wrote:
> doing the longer one, will report back
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair --init-extent-tree /dev/mapper/dshelf1

Still running, it would be lovely to have a percentage of completion on
this one too :)

Repaired extent references for 331055104
ref mismatch on [331071488 16384] extent item 0, found 1
tree backref 331071488 parent 353419264 not found in extent tree
backpointer mismatch on [331071488 16384]
adding new tree backref on start 331071488 len 16384 parent 353419264 root 353419264
Repaired extent references for 331071488
ref mismatch on [331087872 16384] extent item 0, found 1
tree backref 331087872 parent 353419264 not found in extent tree
backpointer mismatch on [331087872 16384]
adding new tree backref on start 331087872 len 16384 parent 353419264 root 353419264

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-05 15:08                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
@ 2022-05-05 15:27                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  2022-05-06  3:19                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-05 15:27 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, May 5, 2022 at 11:08 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, May 03, 2022 at 10:24:25AM -0700, Marc MERLIN wrote:
> > doing the longer one, will report back
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair --init-extent-tree /dev/mapper/dshelf1
>
> Still running, it would be lovely to have a percentage of completion on
> this one too :)
>
> Repaired extent references for 331055104
> ref mismatch on [331071488 16384] extent item 0, found 1
> tree backref 331071488 parent 353419264 not found in extent tree
> backpointer mismatch on [331071488 16384]
> adding new tree backref on start 331071488 len 16384 parent 353419264 root 353419264
> Repaired extent references for 331071488
> ref mismatch on [331087872 16384] extent item 0, found 1
> tree backref 331087872 parent 353419264 not found in extent tree
> backpointer mismatch on [331087872 16384]
> adding new tree backref on start 331087872 len 16384 parent 353419264 root 353419264
>

Sorry Marc I was busy with the conference and completely misread what
you did.  Cancel the btrfs check now, and then do

btrfs rescue tree-recover <device> // This should succeed without
doing anything but just in case
btrfs rescue init-extent-tree <device> // I'm hoping this will succeed
this time, if not of course tell me
btrfs check --repair <device>

There may be some things left to delete but we're just about there.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-05 15:27                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
@ 2022-05-06  3:19                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  2022-05-07  0:25                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-06  3:19 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, May 05, 2022 at 11:27:32AM -0400, Josef Bacik wrote:
> Sorry Marc I was busy with the conference and completely misread what
> you did.  Cancel the btrfs check now, and then do
> 
> btrfs rescue tree-recover <device> // This should succeed without
> doing anything but just in case
> btrfs rescue init-extent-tree <device> // I'm hoping this will succeed
> this time, if not of course tell me
> btrfs check --repair <device>

Got it. Note that check --repair faile

./gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x55fbfe283bc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55fbfe283bc0
Checking root 2 bytenr 29442048
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 15645018521600
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736
Checking root 11222 bytenr 15645261905920
Checking root 11223 bytenr 13576823635968
Checking root 11224 bytenr 13577126182912
Checking root 159785 bytenr 6781490577408
Checking root 159787 bytenr 15645908385792
Checking root 160494 bytenr 6781245882368
Checking root 160496 bytenr 11822309965824
Checking root 161197 bytenr 6781245865984
Checking root 161199 bytenr 13576850833408
Checking root 162628 bytenr 15645764812800
Checking root 162632 bytenr 6781245898752
Checking root 162645 bytenr 5809981095936
Checking root 163298 bytenr 15645124263936
Checking root 163302 bytenr 6781245915136
Checking root 163303 bytenr 15645018505216
Checking root 163316 bytenr 6781245931520
Checking root 163318 bytenr 15645980491776
Checking root 163916 bytenr 11822437826560
Checking root 163920 bytenr 11970640084992
Checking root 163921 bytenr 11971073802240
Checking root 164620 bytenr 15645434036224
Checking root 164624 bytenr 15645502210048
Checking root 164633 bytenr 15645526884352
Checking root 165098 bytenr 11970640101376
Checking root 165100 bytenr 11970733621248
Checking root 165198 bytenr 12511656394752
Checking root 165200 bytenr 12511677972480
Checking root 165294 bytenr 13576901328896
Checking root 165298 bytenr 13577133326336
Checking root 165299 bytenr 13577191505920
Checking root 18446744073709551607 bytenr 13576823717888
Tree recovery finished, you can run check now


The delete block bit should be automated somehow, it's quite slow and painful to do by hand (hours again here)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819130,108,0 on root 11223
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0027_2020.1_0602_1208.xz

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819130,108,0" -r 11223 /dev/mapper/dshelf1 

Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819131,108,0 on root 11223
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0026_2020.1_0602_1208.xz
cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON `1` triggered, value 1


gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819131,108,0" -r 11223 /dev/mapper/dshelf1 
FS_INFO IS 0x561c57fe8600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x561c57fe8600
parent transid verify failed on 13576823635968 wanted 1619857 found 1638917
parent transid verify failed on 13576823635968 wanted 1619857 found 1638917
parent transid verify failed on 13576823635968 wanted 1619857 found 1638917


processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,0 on root 11223
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xz
cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON `1` triggered, value 1


Ok, so it's the same problem as last time. My corrupt blocks have been lost/uncorrupted.
So I re-ran the same commands as last time:
./btrfs-corrupt-block -d "1819130,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs-corrupt-block -d "1819131,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs-corrupt-block -d "1819135,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs-corrupt-block -d "1819137,108,0" -r 11223 /dev/mapper/dshelf1

Ok, more still, new ones I didn't have to run last time, still all on this new file: rdi_0059_2020.1_0602_1208.xz
processed 1619902464 of 1635549184 possible bytesIgnoring transid failure
processed 1619918848 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819139,108,0 on root 11223
inode ref info failed???

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819139,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819142,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819143,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819144,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819150,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819153,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819154,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819160,108,16318464" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819161,108,0" -r 11223 /dev/mapper/dshelf1

ok, took hours and I ran out of time, more tomorrow
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819161,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819164,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819165,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819167,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819168,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819169,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819169,108,134217728" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819170,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819175,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819176,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819177,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819179,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819183,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819184,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819187,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819189,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819189,108,134217728" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819191,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819193,108,0" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819193,108,24248320" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
./btrfs-corrupt-block -d "1819193,108,55640064" -r 11223 /dev/mapper/dshelf1
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-06  3:19                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
@ 2022-05-07  0:25                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  2022-05-07  1:15                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-07  0:25 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, May 5, 2022 at 11:19 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, May 05, 2022 at 11:27:32AM -0400, Josef Bacik wrote:
> > Sorry Marc I was busy with the conference and completely misread what
> > you did.  Cancel the btrfs check now, and then do
> >
> > btrfs rescue tree-recover <device> // This should succeed without
> > doing anything but just in case
> > btrfs rescue init-extent-tree <device> // I'm hoping this will succeed
> > this time, if not of course tell me
> > btrfs check --repair <device>
>
> Got it. Note that check --repair faile
>
> ./gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> FS_INFO IS 0x55fbfe283bc0
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x55fbfe283bc0
> Checking root 2 bytenr 29442048
> Checking root 4 bytenr 15645196861440
> Checking root 5 bytenr 13577660252160
> Checking root 7 bytenr 15645018521600
> Checking root 9 bytenr 15645878108160
> Checking root 11221 bytenr 13577562996736
> Checking root 11222 bytenr 15645261905920
> Checking root 11223 bytenr 13576823635968
> Checking root 11224 bytenr 13577126182912
> Checking root 159785 bytenr 6781490577408
> Checking root 159787 bytenr 15645908385792
> Checking root 160494 bytenr 6781245882368
> Checking root 160496 bytenr 11822309965824
> Checking root 161197 bytenr 6781245865984
> Checking root 161199 bytenr 13576850833408
> Checking root 162628 bytenr 15645764812800
> Checking root 162632 bytenr 6781245898752
> Checking root 162645 bytenr 5809981095936
> Checking root 163298 bytenr 15645124263936
> Checking root 163302 bytenr 6781245915136
> Checking root 163303 bytenr 15645018505216
> Checking root 163316 bytenr 6781245931520
> Checking root 163318 bytenr 15645980491776
> Checking root 163916 bytenr 11822437826560
> Checking root 163920 bytenr 11970640084992
> Checking root 163921 bytenr 11971073802240
> Checking root 164620 bytenr 15645434036224
> Checking root 164624 bytenr 15645502210048
> Checking root 164633 bytenr 15645526884352
> Checking root 165098 bytenr 11970640101376
> Checking root 165100 bytenr 11970733621248
> Checking root 165198 bytenr 12511656394752
> Checking root 165200 bytenr 12511677972480
> Checking root 165294 bytenr 13576901328896
> Checking root 165298 bytenr 13577133326336
> Checking root 165299 bytenr 13577191505920
> Checking root 18446744073709551607 bytenr 13576823717888
> Tree recovery finished, you can run check now
>
>
> The delete block bit should be automated somehow, it's quite slow and painful to do by hand (hours again here)

Agreed, I'm going to wire something up now.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-07  0:25                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
@ 2022-05-07  1:15                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-05-07 15:39                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-07  1:15 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Fri, May 6, 2022 at 8:25 PM Josef Bacik <josef@toxicpanda.com> wrote:
>
> On Thu, May 5, 2022 at 11:19 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Thu, May 05, 2022 at 11:27:32AM -0400, Josef Bacik wrote:
> > > Sorry Marc I was busy with the conference and completely misread what
> > > you did.  Cancel the btrfs check now, and then do
> > >
> > > btrfs rescue tree-recover <device> // This should succeed without
> > > doing anything but just in case
> > > btrfs rescue init-extent-tree <device> // I'm hoping this will succeed
> > > this time, if not of course tell me
> > > btrfs check --repair <device>
> >
> > Got it. Note that check --repair faile
> >
> > ./gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > FS_INFO IS 0x55fbfe283bc0
> > JOSEF: root 9
> > Couldn't find the last root for 8
> > FS_INFO AFTER IS 0x55fbfe283bc0
> > Checking root 2 bytenr 29442048
> > Checking root 4 bytenr 15645196861440
> > Checking root 5 bytenr 13577660252160
> > Checking root 7 bytenr 15645018521600
> > Checking root 9 bytenr 15645878108160
> > Checking root 11221 bytenr 13577562996736
> > Checking root 11222 bytenr 15645261905920
> > Checking root 11223 bytenr 13576823635968
> > Checking root 11224 bytenr 13577126182912
> > Checking root 159785 bytenr 6781490577408
> > Checking root 159787 bytenr 15645908385792
> > Checking root 160494 bytenr 6781245882368
> > Checking root 160496 bytenr 11822309965824
> > Checking root 161197 bytenr 6781245865984
> > Checking root 161199 bytenr 13576850833408
> > Checking root 162628 bytenr 15645764812800
> > Checking root 162632 bytenr 6781245898752
> > Checking root 162645 bytenr 5809981095936
> > Checking root 163298 bytenr 15645124263936
> > Checking root 163302 bytenr 6781245915136
> > Checking root 163303 bytenr 15645018505216
> > Checking root 163316 bytenr 6781245931520
> > Checking root 163318 bytenr 15645980491776
> > Checking root 163916 bytenr 11822437826560
> > Checking root 163920 bytenr 11970640084992
> > Checking root 163921 bytenr 11971073802240
> > Checking root 164620 bytenr 15645434036224
> > Checking root 164624 bytenr 15645502210048
> > Checking root 164633 bytenr 15645526884352
> > Checking root 165098 bytenr 11970640101376
> > Checking root 165100 bytenr 11970733621248
> > Checking root 165198 bytenr 12511656394752
> > Checking root 165200 bytenr 12511677972480
> > Checking root 165294 bytenr 13576901328896
> > Checking root 165298 bytenr 13577133326336
> > Checking root 165299 bytenr 13577191505920
> > Checking root 18446744073709551607 bytenr 13576823717888
> > Tree recovery finished, you can run check now
> >
> >
> > The delete block bit should be automated somehow, it's quite slow and painful to do by hand (hours again here)
>
> Agreed, I'm going to wire something up now.  Thanks,
>

Ok I pushed something, hopefully it works and you don't have to touch
it anymore?  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-07  1:15                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-05-07 15:39                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-05-07 18:58                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-07 15:39 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Fri, May 06, 2022 at 09:15:58PM -0400, Josef Bacik wrote:
> > > The delete block bit should be automated somehow, it's quite slow and painful to do by hand (hours again here)
> >
> > Agreed, I'm going to wire something up now.  Thanks,
> >
> 
> Ok I pushed something, hopefully it works and you don't have to touch
> it anymore?  Thanks,

Thanks. That seems to almost work and I like the progress percentage,
thanks :)

But now it seems to loop on the same one? I got over 50 of those

searching 11223 for bad extents
prIgnoring transid failure35549184 possible bytes, 99%
processed 1620017152 of 1635549184 possible bytes, 99%
Found an extent we don't have a block group for in the file
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0422_2020.1_0602_1208.xz
Deleting

searching 11223 for bad extents
prIgnoring transid failure35549184 possible bytes, 99%
processed 1620017152 of 1635549184 possible bytes, 99%
Found an extent we don't have a block group for in the file
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0422_2020.1_0602_1208.xz
Deleting

searching 11223 for bad extents
prIgnoring transid failure35549184 possible bytes, 99%
processed 1620017152 of 1635549184 possible bytes, 99%
Found an extent we don't have a block group for in the file
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0422_2020.1_0602_1208.xz
Deleting

searching 11223 for bad extents
prIgnoring transid failure35549184 possible bytes, 99%
processed 1620017152 of 1635549184 possible bytes, 99%
Found an extent we don't have a block group for in the file
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0422_2020.1_0602_1208.xz
Deleting


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-07 15:39                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-05-07 18:58                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-05-07 19:36                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-07 18:58 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sat, May 7, 2022 at 11:39 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Fri, May 06, 2022 at 09:15:58PM -0400, Josef Bacik wrote:
> > > > The delete block bit should be automated somehow, it's quite slow and painful to do by hand (hours again here)
> > >
> > > Agreed, I'm going to wire something up now.  Thanks,
> > >
> >
> > Ok I pushed something, hopefully it works and you don't have to touch
> > it anymore?  Thanks,
>
> Thanks. That seems to almost work and I like the progress percentage,
> thanks :)
>
> But now it seems to loop on the same one? I got over 50 of those
>

Did any of the previous ones succeed?  I hope so and we just have one
misbehaving thing.  I've pushed more debugging, maybe it's a large
file that has a lot of broken extents, in either case it'll tell us
what's going on so I can narrow down the problem.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-07 18:58                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-05-07 19:36                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-05-08 19:45                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-07 19:36 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sat, May 07, 2022 at 02:58:38PM -0400, Josef Bacik wrote:
> Did any of the previous ones succeed?  I hope so and we just have one
> misbehaving thing.  I've pushed more debugging, maybe it's a large
> file that has a lot of broken extents, in either case it'll tell us
> what's going on so I can narrow down the problem.  Thanks,

Ok, the debugging helps, now I can see that it's deleting different
blocks for the same filename.
Before I couldn't quite tell if it was making progress, but now I see
it's deleting new locations. 
Looks like I have 100s of deletions coming up, thanks for automating
this.

Will report back, it looks like it may run for a few hours

Afer that just do a 
check --repair
or some other command options?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-07 19:36                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-05-08 19:45                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  2022-05-08 19:55                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-08 19:45 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sat, May 07, 2022 at 12:36:28PM -0700, Marc MERLIN wrote:
> On Sat, May 07, 2022 at 02:58:38PM -0400, Josef Bacik wrote:
> > Did any of the previous ones succeed?  I hope so and we just have one
> > misbehaving thing.  I've pushed more debugging, maybe it's a large
> > file that has a lot of broken extents, in either case it'll tell us
> > what's going on so I can narrow down the problem.  Thanks,
> 
> Ok, the debugging helps, now I can see that it's deleting different
> blocks for the same filename.
> Before I couldn't quite tell if it was making progress, but now I see
> it's deleting new locations. 
> Looks like I have 100s of deletions coming up, thanks for automating
> this.
> 
> Will report back, it looks like it may run for a few hours
> 
> Afer that just do a 
> check --repair
> or some other command options?

Ok, so I got 33GB of output, so glad the tool is automated and the tool
died eventually.
I'll re-run under gdb, at least it did a lot of work.

searching 165298 for bad extents
processed 108707840 of 108756992 possible bytes, 99%
Found an extent we don't have a block group for in the file
inode ref info failed???
History/Clubbing/20220318_Pure_Markus_Schulz.mp4
Deleting [76600, 108, 111673344] root 13576824389632 path top 13576824389632 top slot 24 leaf 11821927333888 slot 52

searching 165298 for bad extents
processed 108707840 of 108756992 possible bytes, 99%
Found an extent we don't have a block group for in the file
inode ref info failed???
History/Clubbing/20220318_Pure_Markus_Schulz.mp4
Deleting [76600, 108, 228950016] root 13576824406016 path top 13576824406016 top slot 24 leaf 11821927350272 slot 52

searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 18446744073709551607 for bad extents
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 3
Floating point exception

I'll re-run and hopefully it will finish without dying or I'll get a backtrace.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-08 19:45                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
@ 2022-05-08 19:55                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  2022-05-08 20:52                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-08 19:55 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 8, 2022 at 3:45 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sat, May 07, 2022 at 12:36:28PM -0700, Marc MERLIN wrote:
> > On Sat, May 07, 2022 at 02:58:38PM -0400, Josef Bacik wrote:
> > > Did any of the previous ones succeed?  I hope so and we just have one
> > > misbehaving thing.  I've pushed more debugging, maybe it's a large
> > > file that has a lot of broken extents, in either case it'll tell us
> > > what's going on so I can narrow down the problem.  Thanks,
> >
> > Ok, the debugging helps, now I can see that it's deleting different
> > blocks for the same filename.
> > Before I couldn't quite tell if it was making progress, but now I see
> > it's deleting new locations.
> > Looks like I have 100s of deletions coming up, thanks for automating
> > this.
> >
> > Will report back, it looks like it may run for a few hours
> >
> > Afer that just do a
> > check --repair
> > or some other command options?
>
> Ok, so I got 33GB of output, so glad the tool is automated and the tool
> died eventually.
> I'll re-run under gdb, at least it did a lot of work.
>
> searching 165298 for bad extents
> processed 108707840 of 108756992 possible bytes, 99%
> Found an extent we don't have a block group for in the file
> inode ref info failed???
> History/Clubbing/20220318_Pure_Markus_Schulz.mp4
> Deleting [76600, 108, 111673344] root 13576824389632 path top 13576824389632 top slot 24 leaf 11821927333888 slot 52
>
> searching 165298 for bad extents
> processed 108707840 of 108756992 possible bytes, 99%
> Found an extent we don't have a block group for in the file
> inode ref info failed???
> History/Clubbing/20220318_Pure_Markus_Schulz.mp4
> Deleting [76600, 108, 228950016] root 13576824406016 path top 13576824406016 top slot 24 leaf 11821927350272 slot 52
>
> searching 165298 for bad extents
> processed 108756992 of 108756992 possible bytes, 100%
> searching 165299 for bad extents
> processed 75792384 of 75792384 possible bytes, 100%
> searching 18446744073709551607 for bad extents
> processed 16384 of 16384 possible bytes, 100%
> Recording extents for root 3
> Floating point exception
>

Oops that's probably a divide by 0 for my fancy pct thing, I fixed it
up and pushed.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-08 19:55                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
@ 2022-05-08 20:52                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  2022-05-08 21:20                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-08 20:52 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 08, 2022 at 03:55:01PM -0400, Josef Bacik wrote:
> > searching 165298 for bad extents
> > processed 108756992 of 108756992 possible bytes, 100%
> > searching 165299 for bad extents
> > processed 75792384 of 75792384 possible bytes, 100%
> > searching 18446744073709551607 for bad extents
> > processed 16384 of 16384 possible bytes, 100%
> > Recording extents for root 3
> > Floating point exception
> 
> Oops that's probably a divide by 0 for my fancy pct thing, I fixed it
> up and pushed.  Thanks,
> 
Mmmh, got worse? Failed on the first file to delete
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
inode ref info failed???
misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,
    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,
    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48
#1  0x000055555559b1a5 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645169139712)
    at common/extent-cache.c:179
#2  0x0000555555584d33 in set_extent_bits (tree=0x100000060, start=15645169139712, end=15645169139711, bits=bits@entry=1)
    at kernel-shared/extent_io.c:380
#3  0x0000555555584f66 in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x0000555555585949 in set_extent_buffer_dirty (eb=eb@entry=0x55555ebf5bd0) at kernel-shared/extent_io.c:976
#5  0x000055555557bc36 in btrfs_mark_buffer_dirty (eb=eb@entry=0x55555ebf5bd0) at kernel-shared/disk-io.c:2224
#6  0x000055555557ef58 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825108188384, parent=0,
    iref=0xffffffffffffffe3, path=0x55555c3e9ce0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825108188384, parent=0,
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x55555c3e9ce0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x55555ca62310, root=root@entry=0x555555c50fa0, bytenr=<optimized out>,
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=165198, owner=76300, offset=0)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555e05e5 in process_eb (trans=trans@entry=0x55555ca62310, root=root@entry=0x555555c50fa0, eb=eb@entry=0x5555632213b0,
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:869
#10 0x00005555555e0750 in process_eb (trans=trans@entry=0x55555ca62310, root=root@entry=0x555555c50fa0, eb=eb@entry=0x55556321d330,
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:944
#11 0x00005555555e0750 in process_eb (trans=trans@entry=0x55555ca62310, root=root@entry=0x555555c50fa0, eb=0x555556497960,
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:944
#12 0x00005555555e0a00 in record_root (root=0x555555c50fa0) at cmds/rescue-init-extent-tree.c:1020
#13 0x00005555555dfebf in foreach_root (fs_info=fs_info@entry=0x55555564dbc0, cb=cb@entry=0x5555555e08fb <record_root>)
    at cmds/rescue-init-extent-tree.c:87
#14 0x00005555555e0d5b in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:1099
#15 0x00005555555d7b65 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-08 20:52                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
@ 2022-05-08 21:20                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-05-08 21:49                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-08 21:20 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 08, 2022 at 01:52:24PM -0700, Marc MERLIN wrote:
> On Sun, May 08, 2022 at 03:55:01PM -0400, Josef Bacik wrote:
> > > searching 165298 for bad extents
> > > processed 108756992 of 108756992 possible bytes, 100%
> > > searching 165299 for bad extents
> > > processed 75792384 of 75792384 possible bytes, 100%
> > > searching 18446744073709551607 for bad extents
> > > processed 16384 of 16384 possible bytes, 100%
> > > Recording extents for root 3
> > > Floating point exception
> > 
> > Oops that's probably a divide by 0 for my fancy pct thing, I fixed it
> > up and pushed.  Thanks,
> > 
> Mmmh, got worse? Failed on the first file to delete
> processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
> inode ref info failed???
> misc/add0/new/file
> Failed to find [10467695652864, 168, 8675328]
> 
> Program received signal SIGSEGV, Segmentation fault.
> rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,
>     next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48
> 48              struct rb_node *n = root->rb_node;
> (gdb) bt
> #0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,
>     next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48
> #1  0x000055555559b1a5 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645169139712)
>     at common/extent-cache.c:179
> #2  0x0000555555584d33 in set_extent_bits (tree=0x100000060, start=15645169139712, end=15645169139711, bits=bits@entry=1)
>     at kernel-shared/extent_io.c:380

I tried the old code, and here is the output 

processed 108756992 of 108756992 possible bytes, 100% 
searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 18446744073709551607 for bad extents 
processed 16384 of 16384 possible bytes, 100% 
Recording extents for root 3

Program received signal SIGFPE, Arithmetic exception.
0x00005555555e08ad in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=eb@entry=0x555555654c10,
    current=current@entry=0x7fffffffdae8) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:112
112       return __printf_chk (__USE_FORTIFY_LEVEL - 1, __fmt, __va_arg_pack ());
(gdb) bt
#0  0x00005555555e08ad in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=eb@entry=0x555555654c10, 
    current=current@entry=0x7fffffffdae8) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:112
#1  0x00005555555e0745 in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=0x555555650b90, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:941
#2  0x00005555555e09ea in record_root (root=0x55555564e030) at cmds/rescue-init-extent-tree.c:1014
#3  0x00005555555e0d11 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:1086
#4  0x00005555555d7b65 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#5  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#6  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-08 21:20                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-05-08 21:49                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-05-08 22:14                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-08 21:49 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 8, 2022 at 5:20 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 08, 2022 at 01:52:24PM -0700, Marc MERLIN wrote:
> > On Sun, May 08, 2022 at 03:55:01PM -0400, Josef Bacik wrote:
> > > > searching 165298 for bad extents
> > > > processed 108756992 of 108756992 possible bytes, 100%
> > > > searching 165299 for bad extents
> > > > processed 75792384 of 75792384 possible bytes, 100%
> > > > searching 18446744073709551607 for bad extents
> > > > processed 16384 of 16384 possible bytes, 100%
> > > > Recording extents for root 3
> > > > Floating point exception
> > >
> > > Oops that's probably a divide by 0 for my fancy pct thing, I fixed it
> > > up and pushed.  Thanks,
> > >
> > Mmmh, got worse? Failed on the first file to delete
> > processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
> > inode ref info failed???
> > misc/add0/new/file
> > Failed to find [10467695652864, 168, 8675328]
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,
> >     next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48
> > 48              struct rb_node *n = root->rb_node;
> > (gdb) bt
> > #0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,
> >     next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48
> > #1  0x000055555559b1a5 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645169139712)
> >     at common/extent-cache.c:179
> > #2  0x0000555555584d33 in set_extent_bits (tree=0x100000060, start=15645169139712, end=15645169139711, bits=bits@entry=1)
> >     at kernel-shared/extent_io.c:380
>
> I tried the old code, and here is the output
>
> processed 108756992 of 108756992 possible bytes, 100%
> searching 165299 for bad extents
> processed 75792384 of 75792384 possible bytes, 100%
> searching 18446744073709551607 for bad extents
> processed 16384 of 16384 possible bytes, 100%
> Recording extents for root 3
>
> Program received signal SIGFPE, Arithmetic exception.
> 0x00005555555e08ad in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=eb@entry=0x555555654c10,
>     current=current@entry=0x7fffffffdae8) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:112
> 112       return __printf_chk (__USE_FORTIFY_LEVEL - 1, __fmt, __va_arg_pack ());
> (gdb) bt
> #0  0x00005555555e08ad in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=eb@entry=0x555555654c10,
>     current=current@entry=0x7fffffffdae8) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:112
> #1  0x00005555555e0745 in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=0x555555650b90,
>     current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:941
> #2  0x00005555555e09ea in record_root (root=0x55555564e030) at cmds/rescue-init-extent-tree.c:1014
> #3  0x00005555555e0d11 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
>     at cmds/rescue-init-extent-tree.c:1086
> #4  0x00005555555d7b65 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
>     at cmds/rescue.c:65
> #5  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
>     at cmds/commands.h:125
> #6  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
> #7  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125
> #8  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
>

Yeah this is the divide by 0, the error you posted earlier is likely
because of the code refactor I did to make the delete thing work.
I've added some more debugging to see if we're not deleting this
problem bytenr during the search for bad extents.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-08 21:49                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-05-08 22:14                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-05-09  0:22                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-08 22:14 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 08, 2022 at 05:49:17PM -0400, Josef Bacik wrote:
> Yeah this is the divide by 0, the error you posted earlier is likely
> because of the code refactor I did to make the delete thing work.
> I've added some more debugging to see if we're not deleting this
> problem bytenr during the search for bad extents.  Thanks,

inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
searching 4 for bad extents
processed 1032192 of 1064960 possible bytes, 96%
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%WTF IT DIDN'T DELETE IT!!
processed 19365888 of 49479680 possible bytes, 39%WTF IT DIDN'T DELETE IT!!
processed 25067520 of 49479680 possible bytes, 50%WTF IT DIDN'T DELETE IT!!
processed 42336256 of 49479680 possible bytes, 85%WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 254902272 of 1635549184 possible bytes, 15%WTF IT DIDN'T DELETE IT!!
processed 311050240 of 1635549184 possible bytes, 19%WTF IT DIDN'T DELETE IT!!
processed 340901888 of 1635549184 possible bytes, 20%WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358498304 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358514688 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358563840 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358662144 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358711296 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!
processed 358727680 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358744064 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358776832 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!
processed 358793216 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358842368 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358875136 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358891520 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!
processed 358907904 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 359022592 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!

many many lines from the beginning, that I wasn't getting before, so I stopped it 

Keep going?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-08 22:14                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-05-09  0:22                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-05-09  0:46                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-09  0:22 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 8, 2022 at 6:14 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 08, 2022 at 05:49:17PM -0400, Josef Bacik wrote:
> > Yeah this is the divide by 0, the error you posted earlier is likely
> > because of the code refactor I did to make the delete thing work.
> > I've added some more debugging to see if we're not deleting this
> > problem bytenr during the search for bad extents.  Thanks,
>

Ooooh right this is the other problem, overlapping extents.  This is
going to be trickier to work out, I'll start writing it up, but I want
to make it automatic as well, so probably won't have anything until
the morning.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-09  0:22                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-05-09  0:46                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-05-09 16:17                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-09  0:46 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 08, 2022 at 08:22:19PM -0400, Josef Bacik wrote:
> On Sun, May 8, 2022 at 6:14 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sun, May 08, 2022 at 05:49:17PM -0400, Josef Bacik wrote:
> > > Yeah this is the divide by 0, the error you posted earlier is likely
> > > because of the code refactor I did to make the delete thing work.
> > > I've added some more debugging to see if we're not deleting this
> > > problem bytenr during the search for bad extents.  Thanks,
> >
> 
> Ooooh right this is the other problem, overlapping extents.  This is
> going to be trickier to work out, I'll start writing it up, but I want
> to make it automatic as well, so probably won't have anything until
> the morning.  Thanks,

Thanks for the heads up

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-09  0:46                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-05-09 16:17                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-05-09 17:00                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-09 16:17 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 8, 2022 at 8:46 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 08, 2022 at 08:22:19PM -0400, Josef Bacik wrote:
> > On Sun, May 8, 2022 at 6:14 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sun, May 08, 2022 at 05:49:17PM -0400, Josef Bacik wrote:
> > > > Yeah this is the divide by 0, the error you posted earlier is likely
> > > > because of the code refactor I did to make the delete thing work.
> > > > I've added some more debugging to see if we're not deleting this
> > > > problem bytenr during the search for bad extents.  Thanks,
> > >
> >
> > Ooooh right this is the other problem, overlapping extents.  This is
> > going to be trickier to work out, I'll start writing it up, but I want
> > to make it automatic as well, so probably won't have anything until
> > the morning.  Thanks,
>
> Thanks for the heads up
>

Ok I've pushed some code, but I'm sitting in a dealership so testing
was light.  I've added a 10 second pause before doing deletions from
the new code to give you time to spot check if the numbers look
insane.  It'll only do that 5 times, so if everything looks good it'll
just yolo delete stuff as it goes after 5 pauses.  Let me know if you
have trouble, thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-09 16:17                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-05-09 17:00                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-05-09 17:09                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-09 17:00 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, May 09, 2022 at 12:17:30PM -0400, Josef Bacik wrote:
> On Sun, May 8, 2022 at 8:46 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sun, May 08, 2022 at 08:22:19PM -0400, Josef Bacik wrote:
> > > On Sun, May 8, 2022 at 6:14 PM Marc MERLIN <marc@merlins.org> wrote:
> > > >
> > > > On Sun, May 08, 2022 at 05:49:17PM -0400, Josef Bacik wrote:
> > > > > Yeah this is the divide by 0, the error you posted earlier is likely
> > > > > because of the code refactor I did to make the delete thing work.
> > > > > I've added some more debugging to see if we're not deleting this
> > > > > problem bytenr during the search for bad extents.  Thanks,
> > > >
> > >
> > > Ooooh right this is the other problem, overlapping extents.  This is
> > > going to be trickier to work out, I'll start writing it up, but I want
> > > to make it automatic as well, so probably won't have anything until
> > > the morning.  Thanks,
> >
> > Thanks for the heads up
> 
> Ok I've pushed some code, but I'm sitting in a dealership so testing
> was light.  I've added a 10 second pause before doing deletions from
> the new code to give you time to spot check if the numbers look
> insane.  It'll only do that 5 times, so if everything looks good it'll
> just yolo delete stuff as it goes after 5 pauses.  Let me know if you
> have trouble, thanks,

Thanks.
It worked for a while, and then failed in a new way?

inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
searching 4 for bad extents
processed 1032192 of 1064960 possible bytes, 96%
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [123032674304 123032702976] current [123032702976 123032731648]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anythin
g1
2
0
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 151552] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 92

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [123032731648 123032760320] current [123032760320 123032788992]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1
2
3
4
5
6
7
8
9
10
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 413696] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 93

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [128877895680 128877924352] current [128877924352 128877948928]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1
2
3
4
5
6
7
8
9
10
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 675840] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 94

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [128877948928 128877977600] current [128877977600 128878002176]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1
2
3
4
5
6
7
8
9
10
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 937984] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 95

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [128878002176 128878030848] current [128878030848 128878055424]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1
2
3
4
5
6
7
8
9
10
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 1200128] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 96

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [128878055424 128878080000] current [128878080000 128878108672]
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 1462272] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 97

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [147706290176 147706318848] current [147706318848 147706343424]
The newly found extent is older, deleting it
inode ref info failed???
DS1   
Deleting [1003, 108, 1855488] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 99

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [147706343424 147706372096] current [147706372096 147706400768]
The newly found extent is older, deleting it
inode ref info failed???
DS1   
Deleting [1003, 108, 2125824] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 101

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [147706400768 147706429440] current [147706429440 147706458112]
The newly found extent is older, deleting it
inode ref info failed???
DS1   
Deleting [1003, 108, 2387968] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 102

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [128878108672 128878133248] current [128878133248 128878157824]
The newly found extent is older, deleting it
inode ref info failed???
DS1   
Deleting [1003, 108, 2650112] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 103

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [147706458112 147706482688] current [147706482688 147706507264]
The newly found extent is older, deleting it
inode ref info failed???
DS1   
Deleting [1003, 108, 2752512] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 103

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [193839411200 193839439872] current [193839439872 193839468544]
The newly found extent is older, deleting it
inode ref info failed???
DS1   
Deleting [1003, 108, 3145728] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 105

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [193839468544 193839497216] current [193839497216 193839529984]
The newly found extent is older, deleting it
inode ref info failed???
DS1   
Deleting [1003, 108, 3407872] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 106

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [193839529984 193839570944] current [193839570944 193839611904]
The newly found extent is older, deleting it
inode ref info failed???
DS1   
Deleting [1003, 108, 3670016] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 107

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [193839611904 193839656960] current [193839656960 193839665152]
The newly found extent is older, deleting it
inode ref info failed???
DS1   
Deleting [1003, 108, 3932160] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 108

(...)

searching 5 for bad extents
processed 32768 of 10977280 possible bytes, 0%
Found an overlapping extent orig [109405609984 109405650944] current [109405650944 109405659136]
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 16121856] root 13576823717888 path top 13576823717888 top slot 0 leaf 11970640084992 slot 25

searching 5 for bad extents
processed 32768 of 10977280 possible bytes, 0%
Found an overlapping extent orig [153036664832 153036668928] current [153036668928 153036689408]
The original extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 15069184] root 13576823898112 path top 13576823898112 top slot 0 leaf 11970640101376 slot 20

searching 5 for bad extents
processed 32768 of 10977280 possible bytes, 0%
Found an overlapping extent orig [153036664832 153036668928] current [153036668928 153036689408]
The original extent is older, deleting it
inode ref info failed???
DS1
ERROR: error searching for key?? -2

wtf
it failed?? -2
ERROR: failed to clear bad extents
doing close???
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=49152
extent buffer leak: start 13576823717888 len 16384
extent buffer leak: start 13576823717888 len 16384
WARNING: dirty eb leak (aborted trans): start 13576823717888 len 16384
extent buffer leak: start 13576823750656 len 16384
extent buffer leak: start 13576823750656 len 16384
WARNING: dirty eb leak (aborted trans): start 13576823750656 len 16384
extent buffer leak: start 11970640084992 len 16384
extent buffer leak: start 11970640084992 len 16384
WARNING: dirty eb leak (aborted trans): start 11970640084992 len 16384
Init extent tree failed
[Inferior 1 (process 19271) exited with code 0376]

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-09 17:00                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-05-09 17:09                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-05-09 17:19                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-09 17:09 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 9, 2022 at 1:00 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 09, 2022 at 12:17:30PM -0400, Josef Bacik wrote:
> > On Sun, May 8, 2022 at 8:46 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sun, May 08, 2022 at 08:22:19PM -0400, Josef Bacik wrote:
> > > > On Sun, May 8, 2022 at 6:14 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > >
> > > > > On Sun, May 08, 2022 at 05:49:17PM -0400, Josef Bacik wrote:
> > > > > > Yeah this is the divide by 0, the error you posted earlier is likely
> > > > > > because of the code refactor I did to make the delete thing work.
> > > > > > I've added some more debugging to see if we're not deleting this
> > > > > > problem bytenr during the search for bad extents.  Thanks,
> > > > >
> > > >
> > > > Ooooh right this is the other problem, overlapping extents.  This is
> > > > going to be trickier to work out, I'll start writing it up, but I want
> > > > to make it automatic as well, so probably won't have anything until
> > > > the morning.  Thanks,
> > >
> > > Thanks for the heads up
> >
> > Ok I've pushed some code, but I'm sitting in a dealership so testing
> > was light.  I've added a 10 second pause before doing deletions from
> > the new code to give you time to spot check if the numbers look
> > insane.  It'll only do that 5 times, so if everything looks good it'll
> > just yolo delete stuff as it goes after 5 pauses.  Let me know if you
> > have trouble, thanks,
>
> Thanks.
> It worked for a while, and then failed in a new way?
>
> inserting block group 15837217947648
> inserting block group 15838291689472
> inserting block group 15839365431296
> inserting block group 15840439173120
> inserting block group 15842586656768
> searching 4 for bad extents
> processed 1032192 of 1064960 possible bytes, 96%
> searching 5 for bad extents
> processed 16384 of 10977280 possible bytes, 0%
> Found an overlapping extent orig [123032674304 123032702976] current [123032702976 123032731648]
> I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't

Ugh shit, I had an off by one error, that's not great.  I've fixed
that up and adjusted the debugging, lets see how that goes.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-09 17:09                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-05-09 17:19                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-05-10  1:04                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-09 17:19 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, May 09, 2022 at 01:09:37PM -0400, Josef Bacik wrote:
> Ugh shit, I had an off by one error, that's not great.  I've fixed
> that up and adjusted the debugging, lets see how that goes.  Thanks,

Looks better this time. Went pretty far, until here:
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163302
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163303
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163316
processed 933888 of 108691456 possible bytes, 0%
Recording extents for root 163318
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 163916
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163920
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163921
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 164624
processed 98304 of 109445120 possible bytes, 0%
Recording extents for root 164633
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55
55              struct rb_node *n = root->rb_node;
(gdb) bt 
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55
#1  0x000055555559b199 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645171073024)
    at common/extent-cache.c:179
#2  0x0000555555584d27 in set_extent_bits (tree=0x100000060, start=15645171073024, end=15645171073023, bits=bits@entry=1)
    at kernel-shared/extent_io.c:380
#3  0x0000555555584f5a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x000055555558593d in set_extent_buffer_dirty (eb=eb@entry=0x55556ce60d60) at kernel-shared/extent_io.c:976
#5  0x000055555557bc2a in btrfs_mark_buffer_dirty (eb=eb@entry=0x55556ce60d60) at kernel-shared/disk-io.c:2224
#6  0x000055555557ef4c in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93824992431962, parent=0, 
    iref=0xffffffffffffffe3, path=0x5555703388f0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93824992431962, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x5555703388f0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x5555685c9d90, root=root@entry=0x5555563d6bf0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=165198, owner=76300, offset=0)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555e086b in process_eb (trans=trans@entry=0x5555685c9d90, root=root@entry=0x5555563d6bf0, eb=eb@entry=0x555571271c40, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:986
#10 0x00005555555e09d6 in process_eb (trans=trans@entry=0x5555685c9d90, root=root@entry=0x5555563d6bf0, eb=eb@entry=0x55557126dbc0, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1061
#11 0x00005555555e09d6 in process_eb (trans=trans@entry=0x5555685c9d90, root=root@entry=0x5555563d6bf0, eb=0x55555649b860, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1061
#12 0x00005555555e0c82 in record_root (root=0x5555563d6bf0) at cmds/rescue-init-extent-tree.c:1137
#13 0x00005555555dfd9f in foreach_root (fs_info=fs_info@entry=0x55555564dbc0, cb=cb@entry=0x5555555e0b7d <record_root>)
    at cmds/rescue-init-extent-tree.c:151
#14 0x00005555555e0ff4 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:1216
#15 0x00005555555d7b70 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-09 17:19                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-05-10  1:04                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-05-10  1:08                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-10  1:04 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 9, 2022 at 1:19 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 09, 2022 at 01:09:37PM -0400, Josef Bacik wrote:
> > Ugh shit, I had an off by one error, that's not great.  I've fixed
> > that up and adjusted the debugging, lets see how that goes.  Thanks,
>

Sorry my laptop battery died while I was at the dealership, and of
course that took allllll day.  Anyway pushed some debugging, am
confused, hopefully won't be confused long.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-10  1:04                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-05-10  1:08                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-05-10  1:18                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-10  1:08 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, May 09, 2022 at 09:04:36PM -0400, Josef Bacik wrote:
> On Mon, May 9, 2022 at 1:19 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Mon, May 09, 2022 at 01:09:37PM -0400, Josef Bacik wrote:
> > > Ugh shit, I had an off by one error, that's not great.  I've fixed
> > > that up and adjusted the debugging, lets see how that goes.  Thanks,
> >
> 
> Sorry my laptop battery died while I was at the dealership, and of
> course that took allllll day.  Anyway pushed some debugging, am
> confused, hopefully won't be confused long.  Thanks,

Sorry :-/  
Yeah, I bring my power supply in such cases :)

Did you upload?
sauron:/var/local/src/btrfs-progs-josefbacik# git pull
Already up to date.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-10  1:08                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-05-10  1:18                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-05-10  1:32                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-10  1:18 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 9, 2022 at 9:08 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 09, 2022 at 09:04:36PM -0400, Josef Bacik wrote:
> > On Mon, May 9, 2022 at 1:19 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Mon, May 09, 2022 at 01:09:37PM -0400, Josef Bacik wrote:
> > > > Ugh shit, I had an off by one error, that's not great.  I've fixed
> > > > that up and adjusted the debugging, lets see how that goes.  Thanks,
> > >
> >
> > Sorry my laptop battery died while I was at the dealership, and of
> > course that took allllll day.  Anyway pushed some debugging, am
> > confused, hopefully won't be confused long.  Thanks,
>
> Sorry :-/
> Yeah, I bring my power supply in such cases :)
>
> Did you upload?
> sauron:/var/local/src/btrfs-progs-josefbacik# git pull
> Already up to date.
>

Sorry, long day, try it again.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-10  1:18                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-05-10  1:32                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-05-10  2:03                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-10  1:32 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, May 09, 2022 at 09:18:32PM -0400, Josef Bacik wrote:
> On Mon, May 9, 2022 at 9:08 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Mon, May 09, 2022 at 09:04:36PM -0400, Josef Bacik wrote:
> > > On Mon, May 9, 2022 at 1:19 PM Marc MERLIN <marc@merlins.org> wrote:
> > > >
> > > > On Mon, May 09, 2022 at 01:09:37PM -0400, Josef Bacik wrote:
> > > > > Ugh shit, I had an off by one error, that's not great.  I've fixed
> > > > > that up and adjusted the debugging, lets see how that goes.  Thanks,
> > > >
> > >
> > > Sorry my laptop battery died while I was at the dealership, and of
> > > course that took allllll day.  Anyway pushed some debugging, am
> > > confused, hopefully won't be confused long.  Thanks,
> >
> > Sorry :-/
> > Yeah, I bring my power supply in such cases :)
> >
> > Did you upload?
> > sauron:/var/local/src/btrfs-progs-josefbacik# git pull
> > Already up to date.
> 
> Sorry, long day, try it again.  Thanks,

processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55
55              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55
#1  0x000055555559b199 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645363896320)
    at common/extent-cache.c:179
#2  0x0000555555584d27 in set_extent_bits (tree=0x100000060, start=15645363896320, end=15645363896319, bits=bits@entry=1)
    at kernel-shared/extent_io.c:380
#3  0x0000555555584f5a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x000055555558593d in set_extent_buffer_dirty (eb=eb@entry=0x55556fe993b0) at kernel-shared/extent_io.c:976
#5  0x000055555557bc2a in btrfs_mark_buffer_dirty (eb=eb@entry=0x55556fe993b0) at kernel-shared/disk-io.c:2224
#6  0x000055555557ef4c in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825336391296, parent=0, 
    iref=0xffffffffffffffe3, path=0x555569d8b680, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825336391296, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555569d8b680, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x55557058a2f0, root=root@entry=0x555559154170, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=165198, owner=76300, offset=0)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555e087d in process_eb (trans=trans@entry=0x55557058a2f0, root=root@entry=0x555559154170, eb=eb@entry=0x55557411e240, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:986
#10 0x00005555555e09e8 in process_eb (trans=trans@entry=0x55557058a2f0, root=root@entry=0x555559154170, eb=eb@entry=0x55557411a1c0, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1061
#11 0x00005555555e09e8 in process_eb (trans=trans@entry=0x55557058a2f0, root=root@entry=0x555559154170, eb=0x555559218660, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1061
#12 0x00005555555e0c98 in record_root (root=0x555559154170) at cmds/rescue-init-extent-tree.c:1137
#13 0x00005555555dfda6 in foreach_root (fs_info=fs_info@entry=0x55555564dbc0, cb=cb@entry=0x5555555e0b93 <record_root>)
    at cmds/rescue-init-extent-tree.c:151
#14 0x00005555555e100a in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:1216
#15 0x00005555555d7b70 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-10  1:32                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-05-10  2:03                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-05-10  2:19                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-10  2:03 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 9, 2022 at 9:32 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 09, 2022 at 09:18:32PM -0400, Josef Bacik wrote:
> > On Mon, May 9, 2022 at 9:08 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Mon, May 09, 2022 at 09:04:36PM -0400, Josef Bacik wrote:
> > > > On Mon, May 9, 2022 at 1:19 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > >
> > > > > On Mon, May 09, 2022 at 01:09:37PM -0400, Josef Bacik wrote:
> > > > > > Ugh shit, I had an off by one error, that's not great.  I've fixed
> > > > > > that up and adjusted the debugging, lets see how that goes.  Thanks,
> > > > >
> > > >
> > > > Sorry my laptop battery died while I was at the dealership, and of
> > > > course that took allllll day.  Anyway pushed some debugging, am
> > > > confused, hopefully won't be confused long.  Thanks,
> > >
> > > Sorry :-/
> > > Yeah, I bring my power supply in such cases :)
> > >
> > > Did you upload?
> > > sauron:/var/local/src/btrfs-progs-josefbacik# git pull
> > > Already up to date.
> >
> > Sorry, long day, try it again.  Thanks,
>
> processed 49152 of 75792384 possible bytes, 0%
> Recording extents for root 165098
> processed 1015808 of 108756992 possible bytes, 0%
> Recording extents for root 165100
> processed 16384 of 49479680 possible bytes, 0%
> Recording extents for root 165198
> processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
> misc/add0/new/file
> Failed to find [10467695652864, 168, 8675328]

Ugh such a pain, lets try this again,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-10  2:03                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-05-10  2:19                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-05-10 13:21                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-10  2:19 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, May 09, 2022 at 10:03:27PM -0400, Josef Bacik wrote:
> On Mon, May 9, 2022 at 9:32 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Mon, May 09, 2022 at 09:18:32PM -0400, Josef Bacik wrote:
> > > On Mon, May 9, 2022 at 9:08 PM Marc MERLIN <marc@merlins.org> wrote:
> > > >
> > > > On Mon, May 09, 2022 at 09:04:36PM -0400, Josef Bacik wrote:
> > > > > On Mon, May 9, 2022 at 1:19 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > > >
> > > > > > On Mon, May 09, 2022 at 01:09:37PM -0400, Josef Bacik wrote:
> > > > > > > Ugh shit, I had an off by one error, that's not great.  I've fixed
> > > > > > > that up and adjusted the debugging, lets see how that goes.  Thanks,
> > > > > >
> > > > >
> > > > > Sorry my laptop battery died while I was at the dealership, and of
> > > > > course that took allllll day.  Anyway pushed some debugging, am
> > > > > confused, hopefully won't be confused long.  Thanks,
> > > >
> > > > Sorry :-/
> > > > Yeah, I bring my power supply in such cases :)
> > > >
> > > > Did you upload?
> > > > sauron:/var/local/src/btrfs-progs-josefbacik# git pull
> > > > Already up to date.
> > >
> > > Sorry, long day, try it again.  Thanks,
> >
> > processed 49152 of 75792384 possible bytes, 0%
> > Recording extents for root 165098
> > processed 1015808 of 108756992 possible bytes, 0%
> > Recording extents for root 165100
> > processed 16384 of 49479680 possible bytes, 0%
> > Recording extents for root 165198
> > processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
> > misc/add0/new/file
> > Failed to find [10467695652864, 168, 8675328]
> 
> Ugh such a pain, lets try this again,


Looks the same?

gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -6
commit 8e14dcb48ad9fd60c595821fc4ebb5c6a1cfb13f
Author: Josef Bacik <josef@toxicpanda.com>
Date:   Mon May 9 22:02:12 2022 -0400

    add bytes


processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
misc/file
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55
55              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55
#1  0x000055555559b199 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645181853696)
    at common/extent-cache.c:179
#2  0x0000555555584d27 in set_extent_bits (tree=0x100000060, start=15645181853696, end=15645181853695, bits=bits@entry=1)
    at kernel-shared/extent_io.c:380
#3  0x0000555555584f5a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x000055555558593d in set_extent_buffer_dirty (eb=eb@entry=0x55556fbb6da0) at kernel-shared/extent_io.c:976
#5  0x000055555557bc2a in btrfs_mark_buffer_dirty (eb=eb@entry=0x55556fbb6da0) at kernel-shared/disk-io.c:2224
#6  0x000055555557ef4c in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825437307456, parent=0, 
    iref=0xffffffffffffffe3, path=0x55556fdc9240, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825437307456, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x55556fdc9240, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555570857380, root=root@entry=0x55555915f5d0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=165198, owner=76300, offset=0)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555e087a in process_eb (trans=trans@entry=0x555570857380, root=root@entry=0x55555915f5d0, eb=eb@entry=0x5555741a4ff0, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:987
#10 0x00005555555e09e5 in process_eb (trans=trans@entry=0x555570857380, root=root@entry=0x55555915f5d0, eb=eb@entry=0x5555741a0f70, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1062
#11 0x00005555555e09e5 in process_eb (trans=trans@entry=0x555570857380, root=root@entry=0x55555915f5d0, eb=0x555559223b80, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1062
#12 0x00005555555e0c95 in record_root (root=0x55555915f5d0) at cmds/rescue-init-extent-tree.c:1138
#13 0x00005555555dfda6 in foreach_root (fs_info=fs_info@entry=0x55555564dbc0, cb=cb@entry=0x5555555e0b90 <record_root>)
    at cmds/rescue-init-extent-tree.c:152
#14 0x00005555555e1007 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:1217
#15 0x00005555555d7b70 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-10  2:19                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-05-10 13:21                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-05-10 14:37                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-10 13:21 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 9, 2022 at 10:19 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 09, 2022 at 10:03:27PM -0400, Josef Bacik wrote:
> > On Mon, May 9, 2022 at 9:32 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Mon, May 09, 2022 at 09:18:32PM -0400, Josef Bacik wrote:
> > > > On Mon, May 9, 2022 at 9:08 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > >
> > > > > On Mon, May 09, 2022 at 09:04:36PM -0400, Josef Bacik wrote:
> > > > > > On Mon, May 9, 2022 at 1:19 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > > > >
> > > > > > > On Mon, May 09, 2022 at 01:09:37PM -0400, Josef Bacik wrote:
> > > > > > > > Ugh shit, I had an off by one error, that's not great.  I've fixed
> > > > > > > > that up and adjusted the debugging, lets see how that goes.  Thanks,
> > > > > > >
> > > > > >
> > > > > > Sorry my laptop battery died while I was at the dealership, and of
> > > > > > course that took allllll day.  Anyway pushed some debugging, am
> > > > > > confused, hopefully won't be confused long.  Thanks,
> > > > >
> > > > > Sorry :-/
> > > > > Yeah, I bring my power supply in such cases :)
> > > > >
> > > > > Did you upload?
> > > > > sauron:/var/local/src/btrfs-progs-josefbacik# git pull
> > > > > Already up to date.
> > > >
> > > > Sorry, long day, try it again.  Thanks,
> > >
> > > processed 49152 of 75792384 possible bytes, 0%
> > > Recording extents for root 165098
> > > processed 1015808 of 108756992 possible bytes, 0%
> > > Recording extents for root 165100
> > > processed 16384 of 49479680 possible bytes, 0%
> > > Recording extents for root 165198
> > > processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
> > > misc/add0/new/file
> > > Failed to find [10467695652864, 168, 8675328]
> >
> > Ugh such a pain, lets try this again,
>
>
> Looks the same?
>

There's no other debug printing before this?  Can I get the full
output from the run?  If there isn't something really wonky is going
on and I'm confused.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-10 13:21                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-05-10 14:37                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-05-10 15:20                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-10 14:37 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, May 10, 2022 at 09:21:33AM -0400, Josef Bacik wrote:
> On Mon, May 9, 2022 at 10:19 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Mon, May 09, 2022 at 10:03:27PM -0400, Josef Bacik wrote:
> > > On Mon, May 9, 2022 at 9:32 PM Marc MERLIN <marc@merlins.org> wrote:
> > > >
> > > > On Mon, May 09, 2022 at 09:18:32PM -0400, Josef Bacik wrote:
> > > > > On Mon, May 9, 2022 at 9:08 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > > >
> > > > > > On Mon, May 09, 2022 at 09:04:36PM -0400, Josef Bacik wrote:
> > > > > > > On Mon, May 9, 2022 at 1:19 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > > > > >
> > > > > > > > On Mon, May 09, 2022 at 01:09:37PM -0400, Josef Bacik wrote:
> > > > > > > > > Ugh shit, I had an off by one error, that's not great.  I've fixed
> > > > > > > > > that up and adjusted the debugging, lets see how that goes.  Thanks,
> > > > > > > >
> > > > > > >
> > > > > > > Sorry my laptop battery died while I was at the dealership, and of
> > > > > > > course that took allllll day.  Anyway pushed some debugging, am
> > > > > > > confused, hopefully won't be confused long.  Thanks,
> > > > > >
> > > > > > Sorry :-/
> > > > > > Yeah, I bring my power supply in such cases :)
> > > > > >
> > > > > > Did you upload?
> > > > > > sauron:/var/local/src/btrfs-progs-josefbacik# git pull
> > > > > > Already up to date.
> > > > >
> > > > > Sorry, long day, try it again.  Thanks,
> > > >
> > > > processed 49152 of 75792384 possible bytes, 0%
> > > > Recording extents for root 165098
> > > > processed 1015808 of 108756992 possible bytes, 0%
> > > > Recording extents for root 165100
> > > > processed 16384 of 49479680 possible bytes, 0%
> > > > Recording extents for root 165198
> > > > processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
> > > > misc/add0/new/file
> > > > Failed to find [10467695652864, 168, 8675328]
> > >
> > > Ugh such a pain, lets try this again,
> >
> >
> > Looks the same?
> 
> There's no other debug printing before this?  Can I get the full
> output from the run?  If there isn't something really wonky is going
> on and I'm confused.  Thanks,

looking for this?
processed 49152 of 0 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [4075, 108, 0]
Couldn't find any paths for this inode

fuller output:
inserting block group 15831849238528
inserting block group 15832922980352
inserting block group 15833996722176
inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
searching 4 for bad extents
processed 1032192 of 1064960 possible bytes, 96%
searching 5 for bad extents
processed 10960896 of 10977280 possible bytes, 99%
searching 7 for bad extents
processed 16384 of 16545742848 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 11221 for bad extents
processed 16384 of 255983616 possible bytes, 0%
searching 11222 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 11223 for bad extents
prIgnoring transid failure35549184 possible bytes, 99%
processed 1635319808 of 1635549184 possible bytes, 99%
searching 11224 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 159785 for bad extents
processed 108429312 of 108429312 possible bytes, 100%
searching 159787 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 160494 for bad extents
processed 108560384 of 108560384 possible bytes, 100%
searching 160496 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 161197 for bad extents
processed 108544000 of 108544000 possible bytes, 100%
searching 161199 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 162628 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 162632 for bad extents
processed 108691456 of 108691456 possible bytes, 100%
searching 162645 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 163298 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 163302 for bad extents
processed 108691456 of 108691456 possible bytes, 100%
searching 163303 for bad extents
processed 71745536 of 75792384 possible bytes, 94%
searching 163316 for bad extents
processed 108691456 of 108691456 possible bytes, 100%
searching 163318 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 163916 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 163920 for bad extents
processed 108691456 of 108691456 possible bytes, 100%
searching 163921 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 164620 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 164624 for bad extents
processed 102318080 of 109445120 possible bytes, 93%
searching 164633 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 165098 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165100 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 165198 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165200 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 165294 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 18446744073709551607 for bad extents
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 3
processed 1556480 of 0 possible bytes, 0%
Recording extents for root 1
processed 49152 of 0 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [4075, 108, 0]
Couldn't find any paths for this inode
doing an insert that overlaps our bytenr 10467701948416 262144
processed 1474560 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes, 96%
Recording extents for root 5
processed 10960896 of 10977280 possible bytes, 99%
Recording extents for root 7
processed 16384 of 16545742848 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 11221
processed 16384 of 255983616 possible bytes, 0%
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes, 100%
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytes, 99%Ignoring transid failure
processed 1635319808 of 1635549184 possible bytes, 99%
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes, 100%
Recording extents for root 159785
processed 108429312 of 108429312 possible bytes, 100%
Recording extents for root 159787
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 160494
processed 1425408 of 108560384 possible bytes, 1%
Recording extents for root 160496
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 161197
processed 770048 of 108544000 possible bytes, 0%
Recording extents for root 161199
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162632
processed 2441216 of 108691456 possible bytes, 2%
Recording extents for root 162645
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163302
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163303
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163316
processed 933888 of 108691456 possible bytes, 0%
Recording extents for root 163318
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 163916
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163920
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163921
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 164624
processed 98304 of 109445120 possible bytes, 0%
Recording extents for root 164633
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467
695652864 8675328
misc/file
Failed to find [10467695652864, 168, 8675328]
Program received signal SIGSEGV, Segmentation fault.

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-10 14:37                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-05-10 15:20                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-05-10 16:06                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-10 15:20 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, May 10, 2022 at 10:37 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, May 10, 2022 at 09:21:33AM -0400, Josef Bacik wrote:
> > On Mon, May 9, 2022 at 10:19 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Mon, May 09, 2022 at 10:03:27PM -0400, Josef Bacik wrote:
> > > > On Mon, May 9, 2022 at 9:32 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > >
> > > > > On Mon, May 09, 2022 at 09:18:32PM -0400, Josef Bacik wrote:
> > > > > > On Mon, May 9, 2022 at 9:08 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > > > >
> > > > > > > On Mon, May 09, 2022 at 09:04:36PM -0400, Josef Bacik wrote:
> > > > > > > > On Mon, May 9, 2022 at 1:19 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > > > > > >
> > > > > > > > > On Mon, May 09, 2022 at 01:09:37PM -0400, Josef Bacik wrote:
> > > > > > > > > > Ugh shit, I had an off by one error, that's not great.  I've fixed
> > > > > > > > > > that up and adjusted the debugging, lets see how that goes.  Thanks,
> > > > > > > > >
> > > > > > > >
> > > > > > > > Sorry my laptop battery died while I was at the dealership, and of
> > > > > > > > course that took allllll day.  Anyway pushed some debugging, am
> > > > > > > > confused, hopefully won't be confused long.  Thanks,
> > > > > > >
> > > > > > > Sorry :-/
> > > > > > > Yeah, I bring my power supply in such cases :)
> > > > > > >
> > > > > > > Did you upload?
> > > > > > > sauron:/var/local/src/btrfs-progs-josefbacik# git pull
> > > > > > > Already up to date.
> > > > > >
> > > > > > Sorry, long day, try it again.  Thanks,
> > > > >
> > > > > processed 49152 of 75792384 possible bytes, 0%
> > > > > Recording extents for root 165098
> > > > > processed 1015808 of 108756992 possible bytes, 0%
> > > > > Recording extents for root 165100
> > > > > processed 16384 of 49479680 possible bytes, 0%
> > > > > Recording extents for root 165198
> > > > > processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
> > > > > misc/add0/new/file
> > > > > Failed to find [10467695652864, 168, 8675328]
> > > >
> > > > Ugh such a pain, lets try this again,
> > >
> > >
> > > Looks the same?
> >
> > There's no other debug printing before this?  Can I get the full
> > output from the run?  If there isn't something really wonky is going
> > on and I'm confused.  Thanks,
>
> looking for this?
> processed 49152 of 0 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [4075, 108, 0]
> Couldn't find any paths for this inode
>

Yup that was it, now it makes sense, I've fixed it hopefully.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-10 15:20                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-05-10 16:06                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-05-10 16:14                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-10 16:06 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, May 10, 2022 at 11:20:51AM -0400, Josef Bacik wrote:
> > looking for this?
> > processed 49152 of 0 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [4075, 108, 0]
> > Couldn't find any paths for this inode
> >
> 
> Yup that was it, now it makes sense, I've fixed it hopefully.  Thanks,

processed 75792384 of 75792384 possible bytes, 100%
searching 164620 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 164624 for bad extents
processed 102318080 of 109445120 possible bytes, 93%
searching 164633 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 165098 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165100 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 165198 for bad extents
processed 108249088 of 108756992 possible bytes, 99%
Found an overlapping extent orig [10467701948416 10467702210560] current [10467695652864 10467704328192]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1
2
3
4
5
6
7
8
9
10
The original extent is older, deleting it
Couldn't find any paths for this inode
Deleting [4075, 108, 0] root 15645018177536 path top 15645018177536 top slot 3 leaf 6781245898752 slot 66

searching 165198 for bad extents
processed 108249088 of 108756992 possible bytes, 99%
Found an overlapping extent orig [10467701948416 10467702210560] current [10467695652864 10467704328192]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1
2
3
4
5
6
7
8
9
10
The original extent is older, deleting it
Couldn't find any paths for this inode
ERROR: error searching for key?? -2

wtf
it failed?? -2
ERROR: failed to clear bad extents
doing close???
ERROR: commit_root already set when starting transaction
WARNING: reserved space leaked, flag=0x4 bytes_reserved=32768
extent buffer leak: start 15645018177536 len 16384
extent buffer leak: start 15645018783744 len 16384
extent buffer leak: start 15645018783744 len 16384
WARNING: dirty eb leak (aborted trans): start 15645018783744 len 16384
extent buffer leak: start 6781245915136 len 16384
extent buffer leak: start 6781245915136 len 16384
WARNING: dirty eb leak (aborted trans): start 6781245915136 len 16384
Init extent tree failed
[Inferior 1 (process 2234) exited with code 0376]

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-10 16:06                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-05-10 16:14                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-05-10 16:44                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-10 16:14 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, May 10, 2022 at 12:06 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, May 10, 2022 at 11:20:51AM -0400, Josef Bacik wrote:
> > > looking for this?
> > > processed 49152 of 0 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [4075, 108, 0]
> > > Couldn't find any paths for this inode
> > >
> >
> > Yup that was it, now it makes sense, I've fixed it hopefully.  Thanks,
>
> processed 75792384 of 75792384 possible bytes, 100%
> searching 164620 for bad extents
> processed 49479680 of 49479680 possible bytes, 100%
> searching 164624 for bad extents
> processed 102318080 of 109445120 possible bytes, 93%
> searching 164633 for bad extents
> processed 75792384 of 75792384 possible bytes, 100%
> searching 165098 for bad extents
> processed 108756992 of 108756992 possible bytes, 100%
> searching 165100 for bad extents
> processed 49479680 of 49479680 possible bytes, 100%
> searching 165198 for bad extents
> processed 108249088 of 108756992 possible bytes, 99%
> Found an overlapping extent orig [10467701948416 10467702210560] current [10467695652864 10467704328192]
> I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1
> 2
> 3
> 4
> 5
> 6
> 7
> 8
> 9
> 10
> The original extent is older, deleting it
> Couldn't find any paths for this inode
> Deleting [4075, 108, 0] root 15645018177536 path top 15645018177536 top slot 3 leaf 6781245898752 slot 66
>
> searching 165198 for bad extents
> processed 108249088 of 108756992 possible bytes, 99%
> Found an overlapping extent orig [10467701948416 10467702210560] current [10467695652864 10467704328192]
> I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1

Ah duh, we delete it the first time, but then we find another
overlapping extent and we try to delete it again and it's not there.
I've fixed this up, try again please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-10 16:14                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-05-10 16:44                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-05-10 21:15                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-10 16:44 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, May 10, 2022 at 12:14:37PM -0400, Josef Bacik wrote:
> Ah duh, we delete it the first time, but then we find another
> overlapping extent and we try to delete it again and it's not there.
> I've fixed this up, try again please.  Thanks,

good news, it worked! :)

I started a simple ./btrfs check --repair /dev/mapper/dshelf1
did you want me to add options to it or run as is?

doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes, 96%
Recording extents for root 5
processed 10960896 of 10977280 possible bytes, 99%
Recording extents for root 7
processed 16384 of 16545742848 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 11221
processed 16384 of 255983616 possible bytes, 0%
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes, 100%
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytes, 99%Ignoring transid failure
processed 1635319808 of 1635549184 possible bytes, 99%
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes, 100%
Recording extents for root 159785
processed 108429312 of 108429312 possible bytes, 100%
Recording extents for root 159787
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 160494
processed 1425408 of 108560384 possible bytes, 1%
Recording extents for root 160496
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 161197
processed 770048 of 108544000 possible bytes, 0%
Recording extents for root 161199
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162632
processed 2441216 of 108691456 possible bytes, 2%
Recording extents for root 162645
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163302
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163303
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163316
processed 933888 of 108691456 possible bytes, 0%
Recording extents for root 163318
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 163916
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163920
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163921
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 164624
processed 98304 of 109445120 possible bytes, 0%
Recording extents for root 164633
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [76300, 108, 0]
misc/file
doing an insert of the bytenr
doing an insert that overlaps our bytenr 10467695652864 8675328
processed 983040 of 108756992 possible bytes, 0%
Recording extents for root 165200
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165294
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165298
processed 524288 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
misc/file
processed 1015808 of 108756992 possible bytes, 0%
doing block accounting
doing close???
Init extent tree finished, you can run check now

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1
enabling repair mode
WARNING:

        Do not use --repair unless you are advised to do so by a developer
        or an experienced user, and then only after having accepted that no
        fsck can successfully repair all types of filesystem corruption. Eg.
        some software or hardware bugs can fatally damage a volume.
        The operation will start in 10 seconds.
        Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
FS_INFO IS 0x56119a61efd0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x56119a61efd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645954129920 wanted 0x085fc24a found 0x941acdde
checksum verify failed on 15645954195456 wanted 0x6c1c72a4 found 0xb06d5dd9
checksum verify failed on 15645608116224 wanted 0x31706547 found 0x314fad1d
checksum verify failed on 15645262413824 wanted 0x3affbd35 found 0x96d53d25
checksum verify failed on 13577199878144 wanted 0x6e9e8bc6 found 0x61063457
checksum verify failed on 13577399156736 wanted 0x2869b8c7 found 0xbb1119e1
checksum verify failed on 12512437698560 wanted 0xca43b3f8 found 0xd7f6db69
checksum verify failed on 13577503686656 wanted 0xd81b7702 found 0x95a3c9a6
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Fixed 0 roots.
[2/7] checking extents
checksum verify failed on 364637798400 wanted 0xc3ba5144 found 0x30a5fff1
checksum verify failed on 364637962240 wanted 0x91d33e8a found 0x00890be7
checksum verify failed on 11970891186176 wanted 0x6b46adfb found 0xe40e4cbe
checksum verify failed on 12511729680384 wanted 0x41be9fa6 found 0x01f06e9c
checksum verify failed on 12512010420224 wanted 0x365027e7 found 0xba73f602
checksum verify failed on 12512303677440 wanted 0xc08fc7a0 found 0x3cb30242
checksum verify failed on 13576951021568 wanted 0x9511e976 found 0xd8fe35ea
checksum verify failed on 13577013624832 wanted 0x76bbe220 found 0xdfabbbdb
checksum verify failed on 13577013919744 wanted 0xafb37267 found 0x69504e0d
(...)
Device extent[1, 11231377227776, 1073741824] didn't find the relative chunk.
Device extent[1, 11406397145088, 1073741824] didn't find the relative chunk.
Device extent[1, 11416060821504, 1073741824] didn't find the relative chunk.
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
data backref 385889992704 parent 11970901524480 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385889992704 parent 11970901524480 owner 0 offset 0 found 1 wanted 0 back 0x5611a97a3110
incorrect local backref count on 385889992704 root 11222 owner 54869 offset 0 found 0 wanted 1 back 0x5611a6e42180
backref disk bytenr does not match extent record, bytenr=385889992704, ref bytenr=0
backpointer mismatch on [385889992704 4096]
repair deleting extent record: key [385889992704,168,4096]
adding new data backref on 385889992704 parent 11970901524480 owner 0 offset 0 found 1
Repaired extent references for 385889992704
data backref 385900392448 parent 1429184937984 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385900392448 parent 1429184937984 owner 0 offset 0 found 1 wanted 0 back 0x5611a659e210
incorrect local backref count on 385900392448 root 11222 owner 55229 offset 524288 found 0 wanted 1 back 0x5611cd078e60
backref disk bytenr does not match extent record, bytenr=385900392448, ref bytenr=0
backpointer mismatch on [385900392448 4096]
repair deleting extent record: key [385900392448,168,4096]
adding new data backref on 385900392448 parent 1429184937984 owner 0 offset 0 found 1
Repaired extent references for 385900392448
data backref 385900478464 parent 11651792814080 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385900478464 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back 0x5611baa7e720
incorrect local backref count on 385900478464 root 11222 owner 55240 offset 0 found 0 wanted 1 back 0x5611cd079880
backref disk bytenr does not match extent record, bytenr=385900478464, ref bytenr=0
backpointer mismatch on [385900478464 4096]
repair deleting extent record: key [385900478464,168,4096]
adding new data backref on 385900478464 parent 11651792814080 owner 0 offset 0 found 1
Repaired extent references for 385900478464
data backref 385907994624 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385907994624 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6ab0
incorrect local backref count on 385907994624 root 11222 owner 55204 offset 131072 found 0 wanted 1 back 0x5611cd0d05c0
backref disk bytenr does not match extent record, bytenr=385907994624, ref bytenr=0
backpointer mismatch on [385907994624 4096]
repair deleting extent record: key [385907994624,168,4096]
adding new data backref on 385907994624 parent 13577102983168 owner 0 offset 0 found 1
Repaired extent references for 385907994624
data backref 385908449280 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385908449280 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a3690
incorrect local backref count on 385908449280 root 11222 owner 55063 offset 524288 found 0 wanted 1 back 0x5611cd0d21d0
backref disk bytenr does not match extent record, bytenr=385908449280, ref bytenr=0
backpointer mismatch on [385908449280 8192]
repair deleting extent record: key [385908449280,168,8192]
adding new data backref on 385908449280 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385908449280
data backref 385908580352 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385908580352 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a36f0
incorrect local backref count on 385908580352 root 11222 owner 55063 offset 655360 found 0 wanted 1 back 0x5611cd0d2db0
backref disk bytenr does not match extent record, bytenr=385908580352, ref bytenr=0
backpointer mismatch on [385908580352 8192]
repair deleting extent record: key [385908580352,168,8192]
adding new data backref on 385908580352 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385908580352
data backref 385909239808 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385909239808 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6b10
incorrect local backref count on 385909239808 root 11222 owner 55204 offset 262144 found 0 wanted 1 back 0x5611cd0c32f0
backref disk bytenr does not match extent record, bytenr=385909239808, ref bytenr=0
backpointer mismatch on [385909239808 4096]
repair deleting extent record: key [385909239808,168,4096]
adding new data backref on 385909239808 parent 13577102983168 owner 0 offset 0 found 1
Repaired extent references for 385909239808
data backref 385913561088 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385913561088 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a3750
incorrect local backref count on 385913561088 root 11222 owner 55063 offset 786432 found 0 wanted 1 back 0x5611cd1196b0
backref disk bytenr does not match extent record, bytenr=385913561088, ref bytenr=0
backpointer mismatch on [385913561088 8192]
repair deleting extent record: key [385913561088,168,8192]
adding new data backref on 385913561088 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385913561088
data backref 385913696256 parent 11651792814080 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385913696256 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back 0x5611baa7e780
incorrect local backref count on 385913696256 root 11222 owner 55240 offset 131072 found 0 wanted 1 back 0x5611cd11a3d0
backref disk bytenr does not match extent record, bytenr=385913696256, ref bytenr=0
backpointer mismatch on [385913696256 4096]
repair deleting extent record: key [385913696256,168,4096]
adding new data backref on 385913696256 parent 11651792814080 owner 0 offset 0 found 1
Repaired extent references for 385913696256
data backref 385914368000 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385914368000 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6a50
incorrect local backref count on 385914368000 root 11222 owner 55204 offset 0 found 0 wanted 1 back 0x5611cd11d660
backref disk bytenr does not match extent record, bytenr=385914368000, ref bytenr=0
backpointer mismatch on [385914368000 8192]
repair deleting extent record: key [385914368000,168,8192]
adding new data backref on 385914368000 parent 13577102983168 owner 0 offset 0 found 1


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-10 16:44                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-05-10 21:15                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  2022-05-10 23:38                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-10 21:15 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

I have
./btrfs check --repair /dev/mapper/dshelf1
on hold (^Z), waiting for your ok to proceed.

All good to continue?

On Tue, May 10, 2022 at 09:44:48AM -0700, Marc MERLIN wrote:
> On Tue, May 10, 2022 at 12:14:37PM -0400, Josef Bacik wrote:
> > Ah duh, we delete it the first time, but then we find another
> > overlapping extent and we try to delete it again and it's not there.
> > I've fixed this up, try again please.  Thanks,
> 
> good news, it worked! :)
> 
> I started a simple ./btrfs check --repair /dev/mapper/dshelf1
> did you want me to add options to it or run as is?
> 
> doing roots
> Recording extents for root 4
> processed 1032192 of 1064960 possible bytes, 96%
> Recording extents for root 5
> processed 10960896 of 10977280 possible bytes, 99%
> Recording extents for root 7
> processed 16384 of 16545742848 possible bytes, 0%
> Recording extents for root 9
> processed 16384 of 16384 possible bytes, 100%
> Recording extents for root 11221
> processed 16384 of 255983616 possible bytes, 0%
> Recording extents for root 11222
> processed 49479680 of 49479680 possible bytes, 100%
> Recording extents for root 11223
> processed 1619902464 of 1635549184 possible bytes, 99%Ignoring transid failure
> processed 1635319808 of 1635549184 possible bytes, 99%
> Recording extents for root 11224
> processed 75792384 of 75792384 possible bytes, 100%
> Recording extents for root 159785
> processed 108429312 of 108429312 possible bytes, 100%
> Recording extents for root 159787
> processed 49152 of 49479680 possible bytes, 0%
> Recording extents for root 160494
> processed 1425408 of 108560384 possible bytes, 1%
> Recording extents for root 160496
> processed 49152 of 49479680 possible bytes, 0%
> Recording extents for root 161197
> processed 770048 of 108544000 possible bytes, 0%
> Recording extents for root 161199
> processed 49152 of 49479680 possible bytes, 0%
> Recording extents for root 162628
> processed 49152 of 49479680 possible bytes, 0%
> Recording extents for root 162632
> processed 2441216 of 108691456 possible bytes, 2%
> Recording extents for root 162645
> processed 49152 of 75792384 possible bytes, 0%
> Recording extents for root 163298
> processed 49152 of 49479680 possible bytes, 0%
> Recording extents for root 163302
> processed 966656 of 108691456 possible bytes, 0%
> Recording extents for root 163303
> processed 49152 of 75792384 possible bytes, 0%
> Recording extents for root 163316
> processed 933888 of 108691456 possible bytes, 0%
> Recording extents for root 163318
> processed 16384 of 49479680 possible bytes, 0%
> Recording extents for root 163916
> processed 49152 of 49479680 possible bytes, 0%
> Recording extents for root 163920
> processed 966656 of 108691456 possible bytes, 0%
> Recording extents for root 163921
> processed 49152 of 75792384 possible bytes, 0%
> Recording extents for root 164620
> processed 49152 of 49479680 possible bytes, 0%
> Recording extents for root 164624
> processed 98304 of 109445120 possible bytes, 0%
> Recording extents for root 164633
> processed 49152 of 75792384 possible bytes, 0%
> Recording extents for root 165098
> processed 1015808 of 108756992 possible bytes, 0%
> Recording extents for root 165100
> processed 16384 of 49479680 possible bytes, 0%
> Recording extents for root 165198
> processed 491520 of 108756992 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [76300, 108, 0]
> misc/file
> doing an insert of the bytenr
> doing an insert that overlaps our bytenr 10467695652864 8675328
> processed 983040 of 108756992 possible bytes, 0%
> Recording extents for root 165200
> processed 16384 of 49479680 possible bytes, 0%
> Recording extents for root 165294
> processed 16384 of 49479680 possible bytes, 0%
> Recording extents for root 165298
> processed 524288 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
> misc/file
> processed 1015808 of 108756992 possible bytes, 0%
> doing block accounting
> doing close???
> Init extent tree finished, you can run check now
> 
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1
> enabling repair mode
> WARNING:
> 
>         Do not use --repair unless you are advised to do so by a developer
>         or an experienced user, and then only after having accepted that no
>         fsck can successfully repair all types of filesystem corruption. Eg.
>         some software or hardware bugs can fatally damage a volume.
>         The operation will start in 10 seconds.
>         Use Ctrl-C to stop it.
> 10 9 8 7 6 5 4 3 2 1
> Starting repair.
> Opening filesystem to check...
> FS_INFO IS 0x56119a61efd0
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x56119a61efd0
> Checking filesystem on /dev/mapper/dshelf1
> UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
> [1/7] checking root items
> checksum verify failed on 15645954129920 wanted 0x085fc24a found 0x941acdde
> checksum verify failed on 15645954195456 wanted 0x6c1c72a4 found 0xb06d5dd9
> checksum verify failed on 15645608116224 wanted 0x31706547 found 0x314fad1d
> checksum verify failed on 15645262413824 wanted 0x3affbd35 found 0x96d53d25
> checksum verify failed on 13577199878144 wanted 0x6e9e8bc6 found 0x61063457
> checksum verify failed on 13577399156736 wanted 0x2869b8c7 found 0xbb1119e1
> checksum verify failed on 12512437698560 wanted 0xca43b3f8 found 0xd7f6db69
> checksum verify failed on 13577503686656 wanted 0xd81b7702 found 0x95a3c9a6
> checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
> checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf
> checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
> checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
> checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
> Fixed 0 roots.
> [2/7] checking extents
> checksum verify failed on 364637798400 wanted 0xc3ba5144 found 0x30a5fff1
> checksum verify failed on 364637962240 wanted 0x91d33e8a found 0x00890be7
> checksum verify failed on 11970891186176 wanted 0x6b46adfb found 0xe40e4cbe
> checksum verify failed on 12511729680384 wanted 0x41be9fa6 found 0x01f06e9c
> checksum verify failed on 12512010420224 wanted 0x365027e7 found 0xba73f602
> checksum verify failed on 12512303677440 wanted 0xc08fc7a0 found 0x3cb30242
> checksum verify failed on 13576951021568 wanted 0x9511e976 found 0xd8fe35ea
> checksum verify failed on 13577013624832 wanted 0x76bbe220 found 0xdfabbbdb
> checksum verify failed on 13577013919744 wanted 0xafb37267 found 0x69504e0d
> (...)
> Device extent[1, 11231377227776, 1073741824] didn't find the relative chunk.
> Device extent[1, 11406397145088, 1073741824] didn't find the relative chunk.
> Device extent[1, 11416060821504, 1073741824] didn't find the relative chunk.
> Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
> Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
> data backref 385889992704 parent 11970901524480 owner 0 offset 0 num_refs 0 not found in extent tree
> incorrect local backref count on 385889992704 parent 11970901524480 owner 0 offset 0 found 1 wanted 0 back 0x5611a97a3110
> incorrect local backref count on 385889992704 root 11222 owner 54869 offset 0 found 0 wanted 1 back 0x5611a6e42180
> backref disk bytenr does not match extent record, bytenr=385889992704, ref bytenr=0
> backpointer mismatch on [385889992704 4096]
> repair deleting extent record: key [385889992704,168,4096]
> adding new data backref on 385889992704 parent 11970901524480 owner 0 offset 0 found 1
> Repaired extent references for 385889992704
> data backref 385900392448 parent 1429184937984 owner 0 offset 0 num_refs 0 not found in extent tree
> incorrect local backref count on 385900392448 parent 1429184937984 owner 0 offset 0 found 1 wanted 0 back 0x5611a659e210
> incorrect local backref count on 385900392448 root 11222 owner 55229 offset 524288 found 0 wanted 1 back 0x5611cd078e60
> backref disk bytenr does not match extent record, bytenr=385900392448, ref bytenr=0
> backpointer mismatch on [385900392448 4096]
> repair deleting extent record: key [385900392448,168,4096]
> adding new data backref on 385900392448 parent 1429184937984 owner 0 offset 0 found 1
> Repaired extent references for 385900392448
> data backref 385900478464 parent 11651792814080 owner 0 offset 0 num_refs 0 not found in extent tree
> incorrect local backref count on 385900478464 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back 0x5611baa7e720
> incorrect local backref count on 385900478464 root 11222 owner 55240 offset 0 found 0 wanted 1 back 0x5611cd079880
> backref disk bytenr does not match extent record, bytenr=385900478464, ref bytenr=0
> backpointer mismatch on [385900478464 4096]
> repair deleting extent record: key [385900478464,168,4096]
> adding new data backref on 385900478464 parent 11651792814080 owner 0 offset 0 found 1
> Repaired extent references for 385900478464
> data backref 385907994624 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree
> incorrect local backref count on 385907994624 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6ab0
> incorrect local backref count on 385907994624 root 11222 owner 55204 offset 131072 found 0 wanted 1 back 0x5611cd0d05c0
> backref disk bytenr does not match extent record, bytenr=385907994624, ref bytenr=0
> backpointer mismatch on [385907994624 4096]
> repair deleting extent record: key [385907994624,168,4096]
> adding new data backref on 385907994624 parent 13577102983168 owner 0 offset 0 found 1
> Repaired extent references for 385907994624
> data backref 385908449280 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree
> incorrect local backref count on 385908449280 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a3690
> incorrect local backref count on 385908449280 root 11222 owner 55063 offset 524288 found 0 wanted 1 back 0x5611cd0d21d0
> backref disk bytenr does not match extent record, bytenr=385908449280, ref bytenr=0
> backpointer mismatch on [385908449280 8192]
> repair deleting extent record: key [385908449280,168,8192]
> adding new data backref on 385908449280 parent 13577747120128 owner 0 offset 0 found 1
> Repaired extent references for 385908449280
> data backref 385908580352 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree
> incorrect local backref count on 385908580352 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a36f0
> incorrect local backref count on 385908580352 root 11222 owner 55063 offset 655360 found 0 wanted 1 back 0x5611cd0d2db0
> backref disk bytenr does not match extent record, bytenr=385908580352, ref bytenr=0
> backpointer mismatch on [385908580352 8192]
> repair deleting extent record: key [385908580352,168,8192]
> adding new data backref on 385908580352 parent 13577747120128 owner 0 offset 0 found 1
> Repaired extent references for 385908580352
> data backref 385909239808 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree
> incorrect local backref count on 385909239808 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6b10
> incorrect local backref count on 385909239808 root 11222 owner 55204 offset 262144 found 0 wanted 1 back 0x5611cd0c32f0
> backref disk bytenr does not match extent record, bytenr=385909239808, ref bytenr=0
> backpointer mismatch on [385909239808 4096]
> repair deleting extent record: key [385909239808,168,4096]
> adding new data backref on 385909239808 parent 13577102983168 owner 0 offset 0 found 1
> Repaired extent references for 385909239808
> data backref 385913561088 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree
> incorrect local backref count on 385913561088 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a3750
> incorrect local backref count on 385913561088 root 11222 owner 55063 offset 786432 found 0 wanted 1 back 0x5611cd1196b0
> backref disk bytenr does not match extent record, bytenr=385913561088, ref bytenr=0
> backpointer mismatch on [385913561088 8192]
> repair deleting extent record: key [385913561088,168,8192]
> adding new data backref on 385913561088 parent 13577747120128 owner 0 offset 0 found 1
> Repaired extent references for 385913561088
> data backref 385913696256 parent 11651792814080 owner 0 offset 0 num_refs 0 not found in extent tree
> incorrect local backref count on 385913696256 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back 0x5611baa7e780
> incorrect local backref count on 385913696256 root 11222 owner 55240 offset 131072 found 0 wanted 1 back 0x5611cd11a3d0
> backref disk bytenr does not match extent record, bytenr=385913696256, ref bytenr=0
> backpointer mismatch on [385913696256 4096]
> repair deleting extent record: key [385913696256,168,4096]
> adding new data backref on 385913696256 parent 11651792814080 owner 0 offset 0 found 1
> Repaired extent references for 385913696256
> data backref 385914368000 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree
> incorrect local backref count on 385914368000 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6a50
> incorrect local backref count on 385914368000 root 11222 owner 55204 offset 0 found 0 wanted 1 back 0x5611cd11d660
> backref disk bytenr does not match extent record, bytenr=385914368000, ref bytenr=0
> backpointer mismatch on [385914368000 8192]
> repair deleting extent record: key [385914368000,168,8192]
> adding new data backref on 385914368000 parent 13577102983168 owner 0 offset 0 found 1
> 
> 
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>  
> Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-10 21:15                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
@ 2022-05-10 23:38                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  2022-05-11  0:08                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-10 23:38 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, May 10, 2022 at 5:15 PM Marc MERLIN <marc@merlins.org> wrote:
>
> I have
> ./btrfs check --repair /dev/mapper/dshelf1
> on hold (^Z), waiting for your ok to proceed.
>
> All good to continue?

Hold on I'm looking at the code, I'm very confused, we shouldn't be
finding any extent tree errors at this point.  Let me work out what's
going on.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-10 23:38                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
@ 2022-05-11  0:08                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  2022-05-11  0:28                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-11  0:08 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, May 10, 2022 at 07:38:37PM -0400, Josef Bacik wrote:
> On Tue, May 10, 2022 at 5:15 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > I have
> > ./btrfs check --repair /dev/mapper/dshelf1
> > on hold (^Z), waiting for your ok to proceed.
> >
> > All good to continue?
> 
> Hold on I'm looking at the code, I'm very confused, we shouldn't be
> finding any extent tree errors at this point.  Let me work out what's
> going on.  Thanks,

Ok.

I ran check without repair, at least it completed. 2.5 million lines of output
but I think they mostly look like this

Opening filesystem to check...
FS_INFO IS 0x555b2100de50
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555b2100de50
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645954129920 wanted 0x085fc24a found 0x941acdde
checksum verify failed on 15645954195456 wanted 0x6c1c72a4 found 0xb06d5dd9
checksum verify failed on 15645608116224 wanted 0x31706547 found 0x314fad1d
checksum verify failed on 15645262413824 wanted 0x3affbd35 found 0x96d53d25
checksum verify failed on 13577199878144 wanted 0x6e9e8bc6 found 0x61063457
checksum verify failed on 13577399156736 wanted 0x2869b8c7 found 0xbb1119e1
checksum verify failed on 12512437698560 wanted 0xca43b3f8 found 0xd7f6db69
checksum verify failed on 13577503686656 wanted 0xd81b7702 found 0x95a3c9a6
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
[2/7] checking extents
checksum verify failed on 364637798400 wanted 0xc3ba5144 found 0x30a5fff1
checksum verify failed on 364637962240 wanted 0x91d33e8a found 0x00890be7
checksum verify failed on 11970891186176 wanted 0x6b46adfb found 0xe40e4cbe
(...)
Device extent[1, 11416060821504, 1073741824] didn't find the relative chunk.
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
data backref 743381753856 parent 782895169536 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 743381753856 parent 782895169536 owner 0 offset 0 found 1 wanted 0 back 0x555b2d253630
incorrect local backref count on 743381753856 root 11222 owner 55083 offset 31752192 found 0 wanted 1 back 0x555b2d4b1bf0
backref disk bytenr does not match extent record, bytenr=743381753856, ref bytenr=0
backpointer mismatch on [743381753856 24576]
data backref 743382028288 parent 11651792814080 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 743382028288 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back 0x555b2d3d6b30
incorrect local backref count on 743382028288 root 11222 owner 55241 offset 917504 found 0 wanted 1 back 0x555b2d4b1eb0
backref disk bytenr does not match extent record, bytenr=743382028288, ref bytenr=0
backpointer mismatch on [743382028288 12288]
data backref 743382425600 parent 11822453374976 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 743382425600 parent 11822453374976 owner 0 offset 0 found 1 wanted 0 back 0x555b52540af0
incorrect local backref count on 743382425600 root 11222 owner 55259 offset 0 found 0 wanted 1 back 0x555b2d4b2d20
backref disk bytenr does not match extent record, bytenr=743382425600, ref bytenr=0
(...)
root 11223 inode 1403275 errors 1000, some csum missing
root 11223 inode 1403276 errors 1000, some csum missing
root 11223 inode 1403277 errors 1000, some csum missing
root 11223 inode 1403278 errors 1000, some csum missing
root 11223 inode 1403279 errors 1000, some csum missing
root 11223 inode 1403280 errors 1000, some csum missing
root 11223 inode 1403281 errors 1500, file extent discount, nbytes wrong, some csum missing
Found file extent holes:
        start: 8192, len: 4096
root 11223 inode 1403282 errors 2001, no inode item, link count wrong
        unresolved ref dir 1403235 index 48 namelen 7 name spdif.h filetype 1 errors 4, no inode ref
root 11223 inode 1403283 errors 2001, no inode item, link count wrong
        unresolved ref dir 1403235 index 49 namelen 10 name speyside.c filetype 1 errors 4, no inode ref
root 11223 inode 1403284 errors 2001, no inode item, link count wrong
        unresolved ref dir 1403235 index 50 namelen 11 name tobermory.c filetype 1 errors 4, no inode ref
root 11223 inode 1403285 errors 2001, no inode item, link count wrong
        unresolved ref dir 1402760 index 19 namelen 2 name sh filetype 2 errors 4, no inode ref
root 11223 inode 1403291 errors 3000, some csum missing, link count wrong
        unresolved ref dir 1403285 index 7 namelen 12 name dma-sh7760.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403292 errors 3000, some csum missing, link count wrong
        unresolved ref dir 1403285 index 8 namelen 5 name fsi.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403293 errors 3000, some csum missing, link count wrong
        unresolved ref dir 1403285 index 9 namelen 5 name hac.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403294 errors 3000, some csum missing, link count wrong
        unresolved ref dir 1403285 index 10 namelen 7 name migor.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403295 errors 2000, link count wrong
        unresolved ref dir 1403285 index 11 namelen 13 name sh7760-ac97.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403296 errors 3000, some csum missing, link count wrong
        unresolved ref dir 1403285 index 12 namelen 5 name siu.h filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403297 errors 3000, some csum missing, link count wrong
        unresolved ref dir 1403285 index 13 namelen 9 name siu_dai.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403298 errors 3000, some csum missing, link count wrong
        unresolved ref dir 1403285 index 14 namelen 9 name siu_pcm.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403299 errors 3000, some csum missing, link count wrong
        unresolved ref dir 1403285 index 15 namelen 5 name ssi.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403301 errors 1000, some csum missing
root 11223 inode 1403302 errors 1000, some csum missing
root 11223 inode 1403303 errors 1000, some csum missing
root 11223 inode 1403304 errors 1000, some csum missing
root 11223 inode 1403305 errors 1000, some csum missing
(...)
root 165299 inode 95687 errors 1000, some csum missing
root 165299 inode 95688 errors 1000, some csum missing
root 165299 inode 95689 errors 1000, some csum missing
root 165299 inode 95690 errors 1000, some csum missing
root 165299 inode 95692 errors 1000, some csum missing
root 165299 inode 95697 errors 1000, some csum missing
root 165299 inode 95698 errors 1000, some csum missing
root 165299 inode 95699 errors 3000, some csum missing, link count wrong
        unresolved ref dir 76854 index 28 namelen 23 name gmapsupp_micronesia.img filetype 1 errors 3, no dir item, no dir index
root 165299 inode 95700 errors 1000, some csum missing
root 165299 inode 95701 errors 3000, some csum missing, link count wrong
        unresolved ref dir 76854 index 30 namelen 24 name gmapsupp_osm_NZ_Topo.img filetype 1 errors 3, no dir item, no dir index
root 165299 inode 95702 errors 1000, some csum missing
root 165299 inode 95703 errors 3000, some csum missing, link count wrong
        unresolved ref dir 76854 index 32 namelen 19 name gmapsupp_osm_AK.img filetype 1 errors 3, no dir item, no dir index
root 165299 inode 95704 errors 1000, some csum missing
root 165299 inode 95705 errors 1000, some csum missing
ERROR: errors found in fs roots
found 14180032409600 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 2083700736
total fs tree bytes: 1891270656
total extent tree bytes: 188317696
btree space waste bytes: 334201707
file data blocks allocated: 14814349185024
 referenced 14828266401792


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-11  0:08                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
@ 2022-05-11  0:28                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  2022-05-11  1:48                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-11  0:28 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, May 10, 2022 at 8:08 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, May 10, 2022 at 07:38:37PM -0400, Josef Bacik wrote:
> > On Tue, May 10, 2022 at 5:15 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > I have
> > > ./btrfs check --repair /dev/mapper/dshelf1
> > > on hold (^Z), waiting for your ok to proceed.
> > >
> > > All good to continue?
> >
> > Hold on I'm looking at the code, I'm very confused, we shouldn't be
> > finding any extent tree errors at this point.  Let me work out what's
> > going on.  Thanks,
>
> Ok.
>
> I ran check without repair, at least it completed. 2.5 million lines of output
> but I think they mostly look like this
>

Ok the csum errors I expect.  I've made some changes and pushed them,
re-run the rescue init-extent-tree again.  Run check without --repair.
If it complains about extent backrefs then I fucked up, I just need
like 20ish lines of that output to see what I'm messing up.  If it
only complains about csums then we won, we can run btrfs check
--init-csum-tree, and then --repair.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-11  0:28                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
@ 2022-05-11  1:48                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  2022-05-11 11:43                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-11  1:48 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, May 10, 2022 at 08:28:09PM -0400, Josef Bacik wrote:
> Ok the csum errors I expect.  I've made some changes and pushed them,
> re-run the rescue init-extent-tree again.  Run check without --repair.
> If it complains about extent backrefs then I fucked up, I just need
> like 20ish lines of that output to see what I'm messing up.  If it
> only complains about csums then we won, we can run btrfs check
> --init-csum-tree, and then --repair.  Thanks,

searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 18446744073709551607 for bad extents
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 3
processed 1556480 of 0 possible bytes, 0%
Recording extents for root 1
processed 1474560 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes, 96%
Recording extents for root 5
processed 10960896 of 10977280 possible bytes, 99%
Recording extents for root 7
processed 16384 of 16545742848 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 11221
processed 49152 of 256016384 possible bytes, 0%
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes, 100%
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytes, 99%Ignoring transid failure
processed 1635319808 of 1635549184 possible bytes, 99%
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes, 100%
Recording extents for root 159785
processed 108429312 of 108429312 possible bytes, 100%
Recording extents for root 159787
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 160494
processed 1425408 of 108560384 possible bytes, 1%
Recording extents for root 160496
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 161197
processed 770048 of 108544000 possible bytes, 0%
Recording extents for root 161199
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162632
processed 2441216 of 108691456 possible bytes, 2%
Recording extents for root 162645
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163302
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163303
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163316
processed 933888 of 108691456 possible bytes, 0%
Recording extents for root 163318
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 163916
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163920
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163921
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 164624
processed 98304 of 109445120 possible bytes, 0%
Recording extents for root 164633
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [76300, 108, 0]
misc/file
doing an insert of the bytenr
doing an insert that overlaps our bytenr 10467695652864 8675328
processed 983040 of 108756992 possible bytes, 0%
Recording extents for root 165200
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165294
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165298
processed 524288 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
misc/file
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165299
processed 16384 of 75792384 possible bytes, 0%
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes, 100%
doing block accounting
doing close???
Init extent tree finished, you can run check now


gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1
Opening filesystem to check...
FS_INFO IS 0x55611fc9ee50
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55611fc9ee50
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645954129920 wanted 0x085fc24a found 0x941acdde
checksum verify failed on 15645954195456 wanted 0x6c1c72a4 found 0xb06d5dd9
checksum verify failed on 15645608116224 wanted 0x31706547 found 0x314fad1d
checksum verify failed on 15645262413824 wanted 0x3affbd35 found 0x96d53d25
checksum verify failed on 13577199878144 wanted 0x6e9e8bc6 found 0x61063457
checksum verify failed on 13577399156736 wanted 0x2869b8c7 found 0xbb1119e1
checksum verify failed on 12512437698560 wanted 0xca43b3f8 found 0xd7f6db69
checksum verify failed on 13577503686656 wanted 0xd81b7702 found 0x95a3c9a6
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
[2/7] checking extents
(...)
root 165299 inode 78934 errors 1000, some csum missing
root 165299 inode 78935 errors 1000, some csum missing
root 165299 inode 78936 errors 1000, some csum missing
root 165299 inode 78937 errors 1000, some csum missing
root 165299 inode 78938 errors 1000, some csum missing
root 165299 inode 78939 errors 1000, some csum missing
root 165299 inode 78940 errors 1000, some csum missing
root 165299 inode 78941 errors 1000, some csum missing
root 165299 inode 78942 errors 1000, some csum missing
root 165299 inode 78943 errors 1000, some csum missing
root 165299 inode 78945 errors 1, no inode item
root 165299 inode 78946 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 2 namelen 12 name screwloo.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78947 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 3 namelen 10 name fround.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78948 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 4 namelen 10 name ginkun.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78949 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 5 namelen 12 name cobracom.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78950 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 6 namelen 11 name wseries.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78951 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 7 namelen 9 name youma.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78952 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 8 namelen 11 name mswordu.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78953 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 9 namelen 12 name bijokkog.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78954 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 10 namelen 8 name kaos.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78955 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 11 namelen 10 name toggle.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78956 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 12 namelen 11 name jungler.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78957 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 13 namelen 10 name wallst.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78958 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 14 namelen 11 name bkrtmaq.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78959 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 15 namelen 12 name spcforce.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78960 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 16 namelen 10 name bstars.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78961 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 17 namelen 12 name bionicc2.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78962 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 18 namelen 10 name crater.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78963 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 19 namelen 12 name mjcamera.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78964 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 20 namelen 10 name aurail.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78965 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 21 namelen 12 name pc_radrc.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78966 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 22 namelen 11 name mgakuen.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78967 errors 3000, some csum missing, link count wrong
        unresolved ref dir 78945 index 23 namelen 12 name blueprnj.zip filetype 1 errors 3, no dir item, no dir index
(...)
root 165299 inode 95686 errors 1000, some csum missing
root 165299 inode 95687 errors 1000, some csum missing
root 165299 inode 95688 errors 1000, some csum missing
root 165299 inode 95689 errors 1000, some csum missing
root 165299 inode 95690 errors 1000, some csum missing
root 165299 inode 95692 errors 1000, some csum missing
root 165299 inode 95697 errors 1000, some csum missing
root 165299 inode 95698 errors 1000, some csum missing
root 165299 inode 95699 errors 3000, some csum missing, link count wrong
        unresolved ref dir 76854 index 28 namelen 23 name gmapsupp_micronesia.img filetype 1 errors 3, no dir item, no dir index
root 165299 inode 95700 errors 1000, some csum missing
root 165299 inode 95701 errors 3000, some csum missing, link count wrong
        unresolved ref dir 76854 index 30 namelen 24 name gmapsupp_osm_NZ_Topo.img filetype 1 errors 3, no dir item, no dir index
root 165299 inode 95702 errors 1000, some csum missing
root 165299 inode 95703 errors 3000, some csum missing, link count wrong
        unresolved ref dir 76854 index 32 namelen 19 name gmapsupp_osm_AK.img filetype 1 errors 3, no dir item, no dir index
root 165299 inode 95704 errors 1000, some csum missing
root 165299 inode 95705 errors 1000, some csum missing
ERROR: errors found in fs roots
Opening filesystem to check...
JOSEF: root 9
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
cache and super generation don't match, space cache will be invalidated
found 14180032393216 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 2083684352
total fs tree bytes: 1891270656
total extent tree bytes: 188301312
btree space waste bytes: 334181897
file data blocks allocated: 14814349185024
 referenced 14828266401792
gargamel:/var/local/src/btrfs-
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-11  1:48                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
@ 2022-05-11 11:43                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  2022-05-11 15:03                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-11 11:43 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, May 10, 2022 at 9:48 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, May 10, 2022 at 08:28:09PM -0400, Josef Bacik wrote:
> > Ok the csum errors I expect.  I've made some changes and pushed them,
> > re-run the rescue init-extent-tree again.  Run check without --repair.
> > If it complains about extent backrefs then I fucked up, I just need
> > like 20ish lines of that output to see what I'm messing up.  If it
> > only complains about csums then we won, we can run btrfs check
> > --init-csum-tree, and then --repair.  Thanks,
>
> searching 165298 for bad extents
> processed 108756992 of 108756992 possible bytes, 100%
> searching 165299 for bad extents
> processed 75792384 of 75792384 possible bytes, 100%
> searching 18446744073709551607 for bad extents
> processed 16384 of 16384 possible bytes, 100%
> Recording extents for root 3
> processed 1556480 of 0 possible bytes, 0%
> Recording extents for root 1
> processed 1474560 of 0 possible bytes, 0%
> doing roots
> Recording extents for root 4
> processed 1032192 of 1064960 possible bytes, 96%
> Recording extents for root 5
> processed 10960896 of 10977280 possible bytes, 99%
> Recording extents for root 7
> processed 16384 of 16545742848 possible bytes, 0%
> Recording extents for root 9
> processed 16384 of 16384 possible bytes, 100%
> Recording extents for root 11221
> processed 49152 of 256016384 possible bytes, 0%
> Recording extents for root 11222
> processed 49479680 of 49479680 possible bytes, 100%
> Recording extents for root 11223
> processed 1619902464 of 1635549184 possible bytes, 99%Ignoring transid failure
> processed 1635319808 of 1635549184 possible bytes, 99%
> Recording extents for root 11224
> processed 75792384 of 75792384 possible bytes, 100%
> Recording extents for root 159785
> processed 108429312 of 108429312 possible bytes, 100%
> Recording extents for root 159787
> processed 49152 of 49479680 possible bytes, 0%
> Recording extents for root 160494
> processed 1425408 of 108560384 possible bytes, 1%
> Recording extents for root 160496
> processed 49152 of 49479680 possible bytes, 0%
> Recording extents for root 161197
> processed 770048 of 108544000 possible bytes, 0%
> Recording extents for root 161199
> processed 49152 of 49479680 possible bytes, 0%
> Recording extents for root 162628
> processed 49152 of 49479680 possible bytes, 0%
> Recording extents for root 162632
> processed 2441216 of 108691456 possible bytes, 2%
> Recording extents for root 162645
> processed 49152 of 75792384 possible bytes, 0%
> Recording extents for root 163298
> processed 49152 of 49479680 possible bytes, 0%
> Recording extents for root 163302
> processed 966656 of 108691456 possible bytes, 0%
> Recording extents for root 163303
> processed 49152 of 75792384 possible bytes, 0%
> Recording extents for root 163316
> processed 933888 of 108691456 possible bytes, 0%
> Recording extents for root 163318
> processed 16384 of 49479680 possible bytes, 0%
> Recording extents for root 163916
> processed 49152 of 49479680 possible bytes, 0%
> Recording extents for root 163920
> processed 966656 of 108691456 possible bytes, 0%
> Recording extents for root 163921
> processed 49152 of 75792384 possible bytes, 0%
> Recording extents for root 164620
> processed 49152 of 49479680 possible bytes, 0%
> Recording extents for root 164624
> processed 98304 of 109445120 possible bytes, 0%
> Recording extents for root 164633
> processed 49152 of 75792384 possible bytes, 0%
> Recording extents for root 165098
> processed 1015808 of 108756992 possible bytes, 0%
> Recording extents for root 165100
> processed 16384 of 49479680 possible bytes, 0%
> Recording extents for root 165198
> processed 491520 of 108756992 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [76300, 108, 0]
> misc/file
> doing an insert of the bytenr
> doing an insert that overlaps our bytenr 10467695652864 8675328
> processed 983040 of 108756992 possible bytes, 0%
> Recording extents for root 165200
> processed 16384 of 49479680 possible bytes, 0%
> Recording extents for root 165294
> processed 16384 of 49479680 possible bytes, 0%
> Recording extents for root 165298
> processed 524288 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
> misc/file
> processed 1015808 of 108756992 possible bytes, 0%
> Recording extents for root 165299
> processed 16384 of 75792384 possible bytes, 0%
> Recording extents for root 18446744073709551607
> processed 16384 of 16384 possible bytes, 100%
> doing block accounting
> doing close???
> Init extent tree finished, you can run check now
>
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1
> Opening filesystem to check...
> FS_INFO IS 0x55611fc9ee50
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x55611fc9ee50
> Checking filesystem on /dev/mapper/dshelf1
> UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
> [1/7] checking root items
> checksum verify failed on 15645954129920 wanted 0x085fc24a found 0x941acdde
> checksum verify failed on 15645954195456 wanted 0x6c1c72a4 found 0xb06d5dd9
> checksum verify failed on 15645608116224 wanted 0x31706547 found 0x314fad1d
> checksum verify failed on 15645262413824 wanted 0x3affbd35 found 0x96d53d25
> checksum verify failed on 13577199878144 wanted 0x6e9e8bc6 found 0x61063457
> checksum verify failed on 13577399156736 wanted 0x2869b8c7 found 0xbb1119e1
> checksum verify failed on 12512437698560 wanted 0xca43b3f8 found 0xd7f6db69
> checksum verify failed on 13577503686656 wanted 0xd81b7702 found 0x95a3c9a6
> checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
> checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf
> checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
> checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
> checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
> [2/7] checking extents

OMG we're almost done, that's awesome.  So no extent errors, just the
csum things and the fs things I expected.  Now we do

btrfs check --init-csum-tree <dev>
btrfs check --repair <dev>

and then you should be good.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-11 11:43                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
@ 2022-05-11 15:03                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
  2022-05-11 15:21                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-11 15:03 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, May 11, 2022 at 07:43:29AM -0400, Josef Bacik wrote:
> OMG we're almost done, that's awesome.  So no extent errors, just the
> csum things and the fs things I expected.  Now we do
 
Yeah!

> btrfs check --init-csum-tree <dev>

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-csum-tree /dev/mapper/dshelf1
Creating a new CRC tree
WARNING:

        Do not use --repair unless you are advised to do so by a developer
        or an experienced user, and then only after having accepted that no
        fsck can successfully repair all types of filesystem corruption. Eg.
        some software or hardware bugs can fatally damage a volume.
        The operation will start in 10 seconds.
        Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
FS_INFO IS 0x55a89d2dbfd0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55a89d2dbfd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Reinitialize checksum tree
checksum verify failed on 42565632 wanted 0x53c18aa1 found 0xcd4368c2
checksum verify failed on 42582016 wanted 0xc42fc10c found 0xd63d1182
checksum verify failed on 42647552 wanted 0x10aefde4 found 0x5c5bedba
checksum verify failed on 42680320 wanted 0x9cfe6b48 found 0xed48fcc5
checksum verify failed on 42696704 wanted 0x005c4793 found 0x0d9de33c
checksum verify failed on 42811392 wanted 0x2b08fbc0 found 0xf25092bb
checksum verify failed on 42827776 wanted 0x6277c597 found 0x70d260d3
checksum verify failed on 42844160 wanted 0xceecf694 found 0x15ab7b7c
checksum verify failed on 42860544 wanted 0xbc48b372 found 0x99719113
checksum verify failed on 60915712 wanted 0xb9ea7152 found 0x75dbb3c0
checksum verify failed on 63438848 wanted 0x4cfafe67 found 0xf97194d0
checksum verify failed on 63422464 wanted 0xcae61e1f found 0xafc5d21c
checksum verify failed on 63946752 wanted 0xe545e347 found 0x5c2ca453
checksum verify failed on 63881216 wanted 0x4dbaba12 found 0x843eeecc
checksum verify failed on 63930368 wanted 0x03a5dd54 found 0x049161ac
checksum verify failed on 76709888 wanted 0x52911556 found 0x1fbf8f99
checksum verify failed on 76660736 wanted 0xcf5d93fa found 0x08ef5ce9
checksum verify failed on 76677120 wanted 0xf56b664d found 0x9d0df1da
checksum verify failed on 76726272 wanted 0x759a69db found 0xc4d2e554
checksum verify failed on 76742656 wanted 0xe5a5401d found 0xd40609cc
checksum verify failed on 100302848 wanted 0xb1829326 found 0xbd08c95e
checksum verify failed on 105742336 wanted 0x26ef0666 found 0x2fa5587b
checksum verify failed on 129449984 wanted 0x915d087e found 0x26b36e0f
ERROR: checksum tree refilling failed: -2

easy one to fix?

> btrfs check --repair <dev>
> 
> and then you should be good.  Thanks,
> 
> Josef
> 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-11 15:03                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
@ 2022-05-11 15:21                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  2022-05-11 16:00                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-11 15:21 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, May 11, 2022 at 11:03 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, May 11, 2022 at 07:43:29AM -0400, Josef Bacik wrote:
> > OMG we're almost done, that's awesome.  So no extent errors, just the
> > csum things and the fs things I expected.  Now we do
>
> Yeah!
>
> > btrfs check --init-csum-tree <dev>
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-csum-tree /dev/mapper/dshelf1
> Creating a new CRC tree
> WARNING:
>
>         Do not use --repair unless you are advised to do so by a developer
>         or an experienced user, and then only after having accepted that no
>         fsck can successfully repair all types of filesystem corruption. Eg.
>         some software or hardware bugs can fatally damage a volume.
>         The operation will start in 10 seconds.
>         Use Ctrl-C to stop it.
> 10 9 8 7 6 5 4 3 2 1
> Starting repair.
> Opening filesystem to check...
> FS_INFO IS 0x55a89d2dbfd0
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x55a89d2dbfd0
> Checking filesystem on /dev/mapper/dshelf1
> UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
> Reinitialize checksum tree
> checksum verify failed on 42565632 wanted 0x53c18aa1 found 0xcd4368c2
> checksum verify failed on 42582016 wanted 0xc42fc10c found 0xd63d1182
> checksum verify failed on 42647552 wanted 0x10aefde4 found 0x5c5bedba
> checksum verify failed on 42680320 wanted 0x9cfe6b48 found 0xed48fcc5
> checksum verify failed on 42696704 wanted 0x005c4793 found 0x0d9de33c
> checksum verify failed on 42811392 wanted 0x2b08fbc0 found 0xf25092bb
> checksum verify failed on 42827776 wanted 0x6277c597 found 0x70d260d3
> checksum verify failed on 42844160 wanted 0xceecf694 found 0x15ab7b7c
> checksum verify failed on 42860544 wanted 0xbc48b372 found 0x99719113
> checksum verify failed on 60915712 wanted 0xb9ea7152 found 0x75dbb3c0
> checksum verify failed on 63438848 wanted 0x4cfafe67 found 0xf97194d0
> checksum verify failed on 63422464 wanted 0xcae61e1f found 0xafc5d21c
> checksum verify failed on 63946752 wanted 0xe545e347 found 0x5c2ca453
> checksum verify failed on 63881216 wanted 0x4dbaba12 found 0x843eeecc
> checksum verify failed on 63930368 wanted 0x03a5dd54 found 0x049161ac
> checksum verify failed on 76709888 wanted 0x52911556 found 0x1fbf8f99
> checksum verify failed on 76660736 wanted 0xcf5d93fa found 0x08ef5ce9
> checksum verify failed on 76677120 wanted 0xf56b664d found 0x9d0df1da
> checksum verify failed on 76726272 wanted 0x759a69db found 0xc4d2e554
> checksum verify failed on 76742656 wanted 0xe5a5401d found 0xd40609cc
> checksum verify failed on 100302848 wanted 0xb1829326 found 0xbd08c95e
> checksum verify failed on 105742336 wanted 0x26ef0666 found 0x2fa5587b
> checksum verify failed on 129449984 wanted 0x915d087e found 0x26b36e0f
> ERROR: checksum tree refilling failed: -2
>

Hmm of course IDK where it's falling over, I pushed some debugging but
there's another method we can try if I can't figure this out.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-11 15:21                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
@ 2022-05-11 16:00                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  2022-05-11 16:05                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-11 16:00 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, May 11, 2022 at 11:21:37AM -0400, Josef Bacik wrote:
> Hmm of course IDK where it's falling over, I pushed some debugging but
> there's another method we can try if I can't figure this out.  Thanks,

Couldn't find the last root for 8
FS_INFO AFTER IS 0x564b2ca91fd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Reinitialize checksum tree
checksum verify failed on 42565632 wanted 0x53c18aa1 found 0xcd4368c2
checksum verify failed on 42582016 wanted 0xc42fc10c found 0xd63d1182
checksum verify failed on 42647552 wanted 0x10aefde4 found 0x5c5bedba
checksum verify failed on 42680320 wanted 0x9cfe6b48 found 0xed48fcc5
checksum verify failed on 42696704 wanted 0x005c4793 found 0x0d9de33c
checksum verify failed on 42811392 wanted 0x2b08fbc0 found 0xf25092bb
checksum verify failed on 42827776 wanted 0x6277c597 found 0x70d260d3
checksum verify failed on 42844160 wanted 0xceecf694 found 0x15ab7b7c
checksum verify failed on 42860544 wanted 0xbc48b372 found 0x99719113
checksum verify failed on 60915712 wanted 0xb9ea7152 found 0x75dbb3c0
checksum verify failed on 63438848 wanted 0x4cfafe67 found 0xf97194d0
checksum verify failed on 63422464 wanted 0xcae61e1f found 0xafc5d21c
checksum verify failed on 63946752 wanted 0xe545e347 found 0x5c2ca453
checksum verify failed on 63881216 wanted 0x4dbaba12 found 0x843eeecc
checksum verify failed on 63930368 wanted 0x03a5dd54 found 0x049161ac
checksum verify failed on 76709888 wanted 0x52911556 found 0x1fbf8f99
checksum verify failed on 76660736 wanted 0xcf5d93fa found 0x08ef5ce9
checksum verify failed on 76677120 wanted 0xf56b664d found 0x9d0df1da
checksum verify failed on 76726272 wanted 0x759a69db found 0xc4d2e554
checksum verify failed on 76742656 wanted 0xe5a5401d found 0xd40609cc
checksum verify failed on 100302848 wanted 0xb1829326 found 0xbd08c95e
checksum verify failed on 105742336 wanted 0x26ef0666 found 0x2fa5587b
checksum verify failed on 129449984 wanted 0x915d087e found 0x26b36e0f
ERROR: iterate extent inodes failed
ERROR: checksum tree refilling failed: -2

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-11 16:00                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
@ 2022-05-11 16:05                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  2022-05-11 18:00                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Goffredo Baroncelli
                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` (2 more replies)
  0 siblings, 3 replies; 479+ messages in thread
From: Josef Bacik @ 2022-05-11 16:05 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, May 11, 2022 at 12:00 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, May 11, 2022 at 11:21:37AM -0400, Josef Bacik wrote:
> > Hmm of course IDK where it's falling over, I pushed some debugging but
> > there's another method we can try if I can't figure this out.  Thanks,
>
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x564b2ca91fd0
> Checking filesystem on /dev/mapper/dshelf1
> UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
> Reinitialize checksum tree
> checksum verify failed on 42565632 wanted 0x53c18aa1 found 0xcd4368c2
> checksum verify failed on 42582016 wanted 0xc42fc10c found 0xd63d1182
> checksum verify failed on 42647552 wanted 0x10aefde4 found 0x5c5bedba
> checksum verify failed on 42680320 wanted 0x9cfe6b48 found 0xed48fcc5
> checksum verify failed on 42696704 wanted 0x005c4793 found 0x0d9de33c
> checksum verify failed on 42811392 wanted 0x2b08fbc0 found 0xf25092bb
> checksum verify failed on 42827776 wanted 0x6277c597 found 0x70d260d3
> checksum verify failed on 42844160 wanted 0xceecf694 found 0x15ab7b7c
> checksum verify failed on 42860544 wanted 0xbc48b372 found 0x99719113
> checksum verify failed on 60915712 wanted 0xb9ea7152 found 0x75dbb3c0
> checksum verify failed on 63438848 wanted 0x4cfafe67 found 0xf97194d0
> checksum verify failed on 63422464 wanted 0xcae61e1f found 0xafc5d21c
> checksum verify failed on 63946752 wanted 0xe545e347 found 0x5c2ca453
> checksum verify failed on 63881216 wanted 0x4dbaba12 found 0x843eeecc
> checksum verify failed on 63930368 wanted 0x03a5dd54 found 0x049161ac
> checksum verify failed on 76709888 wanted 0x52911556 found 0x1fbf8f99
> checksum verify failed on 76660736 wanted 0xcf5d93fa found 0x08ef5ce9
> checksum verify failed on 76677120 wanted 0xf56b664d found 0x9d0df1da
> checksum verify failed on 76726272 wanted 0x759a69db found 0xc4d2e554
> checksum verify failed on 76742656 wanted 0xe5a5401d found 0xd40609cc
> checksum verify failed on 100302848 wanted 0xb1829326 found 0xbd08c95e
> checksum verify failed on 105742336 wanted 0x26ef0666 found 0x2fa5587b
> checksum verify failed on 129449984 wanted 0x915d087e found 0x26b36e0f
> ERROR: iterate extent inodes failed
> ERROR: checksum tree refilling failed: -2
>

Ok that's what I figured, switched to the other method, lets see how
that goes.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-11 16:05                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
@ 2022-05-11 18:00                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Goffredo Baroncelli
  2022-05-12  2:39                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Zygo Blaxell
  2022-05-11 18:58                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
       [not found]                                                                                                                                                                                                                                                                                                                                                                                                                                     ` <20220513144113.GA16501@merlins.org>
  2 siblings, 1 reply; 479+ messages in thread
From: Goffredo Baroncelli @ 2022-05-11 18:00 UTC (permalink / raw)
  To: Josef Bacik, Marc MERLIN; +Cc: linux-btrfs

On 11/05/2022 18.05, Josef Bacik wrote:
> On Wed, May 11, 2022 at 12:00 PM Marc MERLIN <marc@merlins.org> wrote:
>>
>> On Wed, May 11, 2022 at 11:21:37AM -0400, Josef Bacik wrote:
[...]

Hi Joseph, Marc,

I looked back in the thread but I was unable to find if it was discussed.
Even if the size of the FS is quite big, I am wondering if does make sense
that Marc send to Josef the metadata of the filesystem, to speed up the
recover.

I am sure that btrfs-image was considered but then discarded (due to a risk
of leaking of sensible information ? or may be the image would be too big ?);

It would be interested to know the reasons; may be that even if btrfs-image
doesn't fit this particular case in the current form, it can be extended
to handle a case like the Marc one..

BR
-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-11 16:05                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  2022-05-11 18:00                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Goffredo Baroncelli
@ 2022-05-11 18:58                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
       [not found]                                                                                                                                                                                                                                                                                                                                                                                                                                     ` <20220513144113.GA16501@merlins.org>
  2 siblings, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-05-11 18:58 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, May 11, 2022 at 12:05:08PM -0400, Josef Bacik wrote:
> Ok that's what I figured, switched to the other method, lets see how
> that goes.  Thanks,

Seems to work so far
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-csum-tree /dev/mapper/dshelf1
Creating a new CRC tree
WARNING:

        Do not use --repair unless you are advised to do so by a developer
        or an experienced user, and then only after having accepted that no
        fsck can successfully repair all types of filesystem corruption. Eg.
        some software or hardware bugs can fatally damage a volume.
        The operation will start in 10 seconds.
        Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
FS_INFO IS 0x5585f31e5fd0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5585f31e5fd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Reinitialize checksum tree
checksum verify failed on 11970891186176 wanted 0x6b46adfb found 0xe40e4cbe
checksum verify failed on 12511729680384 wanted 0x41be9fa6 found 0x01f06e9c
checksum verify failed on 12512010420224 wanted 0x365027e7 found 0xba73f602
checksum verify failed on 12512303677440 wanted 0xc08fc7a0 found 0x3cb30242
checksum verify failed on 13576951021568 wanted 0x9511e976 found 0xd8fe35ea
checksum verify failed on 13577013624832 wanted 0x76bbe220 found 0xdfabbbdb
(...)

it's taking a while to run, will report when done.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-11 18:00                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Goffredo Baroncelli
@ 2022-05-12  2:39                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Zygo Blaxell
  0 siblings, 0 replies; 479+ messages in thread
From: Zygo Blaxell @ 2022-05-12  2:39 UTC (permalink / raw)
  To: kreijack; +Cc: Josef Bacik, Marc MERLIN, linux-btrfs

On Wed, May 11, 2022 at 08:00:07PM +0200, Goffredo Baroncelli wrote:
> On 11/05/2022 18.05, Josef Bacik wrote:
> > On Wed, May 11, 2022 at 12:00 PM Marc MERLIN <marc@merlins.org> wrote:
> > > 
> > > On Wed, May 11, 2022 at 11:21:37AM -0400, Josef Bacik wrote:
> [...]
> 
> Hi Joseph, Marc,
> 
> I looked back in the thread but I was unable to find if it was discussed.
> Even if the size of the FS is quite big, I am wondering if does make sense
> that Marc send to Josef the metadata of the filesystem, to speed up the
> recover.

> I am sure that btrfs-image was considered but then discarded (due to a risk
> of leaking of sensible information ? or may be the image would be too big ?);

The metadata would be 15GB for the csums alone (according to a
btrfs-dump-super in the thread, only 15TB of the 24TB total space is
used).  Subvol metadata would be on top of that.  Too big for an email,
but not impossible to transfer bidirectionally with a tool such as rsync.

Alternatively, Josef could debug the tool on the copy of the metadata,
then send the tool to Marc to run again on the original metadata, and
unidirectional transfer will suffice.  Handily, this also provides Marc
with a restorable backup copy of the metadata, so if there are bugs in
the recovery tool, the original metadata can be restored to try again
with a fixed tool.

Since the metadata tree is damaged, it can't be traversed (fixing
that is the whole point of this exercise), so btrfs-image won't work.
The image would have to be a raw dump of all the metadata block groups
to get all the potentially relevant pages (including orphan leaf pages
whose parents have been lost due to the disk failures).  In most cases
capturing the free space in metadata block groups would at most double
the size, but 30GB is still manageable.

This is a relatively simple tool to build for the case where the chunk
tree is intact, which it seems to be in this case.  If it wasn't, a
brute-force tool could scan the entire disk and pick up anything that
looks like a metadata page, then write a chunk tree that matches the
majority of collected pages (given a metadata page header's bytenr and
the on-disk position, we can identify which metadata chunk it belongs to;
the first and last page within that chunk provides the logical and
physical boundaries of the chunk).

Dropping the csum pages from the image is possible.  They all have
distinct item keys not found in any other metadata page, so they're easy
to spot on disk and leave out of the transfer; however, the more pages
that aren't part of the image, the more pages that are lost or awkwardly
unverifiable, increasing risk that the recovery tool won't be able to
fix the filesystem.  It's likely that at least the interior nodes of the
csum tree will still need to be part of the recovery operation, if not
all the leaf nodes as well.  The alternative is to blow up the csum tree
and generate a new one from subvol data, but that means no data integrity
checking on any of the data blocks (which are likely also corrupted by
the disk failures).  Undetected corruption becomes inevitable.

Assuming no snapshots and the distribution of file sizes and types
mentioned in the thread (100MB..10GB files, no inline), it would be maybe
10-15 MB for one subvol tree, double that to include the extent tree for
eliminating duplicate or obsolete pages.  That _does_ fit in an email,
especially after compression (subvol trees are highly compressible,
half the bits are zero and a third of the rest are consecutive or
constant numbers).

> It would be interested to know the reasons; may be that even if btrfs-image
> doesn't fit this particular case in the current form, it can be extended
> to handle a case like the Marc one..
> 
> BR
> -- 
> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
       [not found]                                                                                                                                                                                                                                                                                                                                                                                                                                       ` <CAEzrpqfYg=Zf_GYjyvc+WZsnoCjiPTAS-08C_rB=gey74DGUqA@mail.gmail.com>
@ 2022-05-15  2:57                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  2022-05-15 14:02                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-15  2:57 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Fri, May 13, 2022 at 12:16:02PM -0400, Josef Bacik wrote:
> Once Sarah is asleep I'll look at the code, we can probably make this
> go faster, but you've got a lot of data so I expect it's going to take
> some time.  Thanks,

It's still running on my side, almost 4 days. Is there any way to know
whether I'm close to 100%, or not really?

numbers go up and down, so it's not converging towards a bigger number.

checksum verify failed on 12511716491264 wanted 0xb8b7d979 found 0xca46ff1e
checksum verify failed on 12511716409344 wanted 0x67a73a8d found 0xbdc10807
checksum verify failed on 12511716638720 wanted 0x001ed505 found 0x4af8413f
checksum verify failed on 12511724437504 wanted 0x275e729d found 0xe4908156
checksum verify failed on 13577574744064 wanted 0x93a1e209 found 0x29d35de7
checksum verify failed on 12512041566208 wanted 0xe16cec19 found 0xbb3739bd
checksum verify failed on 913014784 wanted 0x4c3d9d32 found 0x561dcfd3
checksum verify failed on 15646027530240 wanted 0x9969c0a6 found 0x7dfb22f6
checksum verify failed on 9227783651328 wanted 0xc6a441d7 found 0x7b82e991
checksum verify failed on 9227783700480 wanted 0x03f1307a found 0xd2d1a1c4
checksum verify failed on 9227817467904 wanted 0x253e13f0 found 0xb2ca98f5
checksum verify failed on 12511748128768 wanted 0x5a02792e found 0x2fd56fdb
(...)
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-15  2:57                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
@ 2022-05-15 14:02                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  2022-05-15 14:41                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-15 14:02 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sat, May 14, 2022 at 10:57 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Fri, May 13, 2022 at 12:16:02PM -0400, Josef Bacik wrote:
> > Once Sarah is asleep I'll look at the code, we can probably make this
> > go faster, but you've got a lot of data so I expect it's going to take
> > some time.  Thanks,
>
> It's still running on my side, almost 4 days. Is there any way to know
> whether I'm close to 100%, or not really?
>

The fs based refill isn't snapshot aware, so it's going to search
everything constantly which is annoying.  I've got some time this
morning, I'll write up something different.  THanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-15 14:02                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
@ 2022-05-15 14:41                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
  2022-05-15 15:24                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-15 14:41 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 15, 2022 at 10:02:10AM -0400, Josef Bacik wrote:
> On Sat, May 14, 2022 at 10:57 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Fri, May 13, 2022 at 12:16:02PM -0400, Josef Bacik wrote:
> > > Once Sarah is asleep I'll look at the code, we can probably make this
> > > go faster, but you've got a lot of data so I expect it's going to take
> > > some time.  Thanks,
> >
> > It's still running on my side, almost 4 days. Is there any way to know
> > whether I'm close to 100%, or not really?
> 
> The fs based refill isn't snapshot aware, so it's going to search
> everything constantly which is annoying.  I've got some time this
> morning, I'll write up something different.  THanks,

Ah, in that case I have around 20 snapshots, so yeah if it's going to be
20 times faster, I'll take that, especially if it ends up being one day
instead of 3 weeks :)

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-15 14:41                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
@ 2022-05-15 15:24                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  2022-05-15 15:33                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-15 15:24 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 15, 2022 at 10:41 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 15, 2022 at 10:02:10AM -0400, Josef Bacik wrote:
> > On Sat, May 14, 2022 at 10:57 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Fri, May 13, 2022 at 12:16:02PM -0400, Josef Bacik wrote:
> > > > Once Sarah is asleep I'll look at the code, we can probably make this
> > > > go faster, but you've got a lot of data so I expect it's going to take
> > > > some time.  Thanks,
> > >
> > > It's still running on my side, almost 4 days. Is there any way to know
> > > whether I'm close to 100%, or not really?
> >
> > The fs based refill isn't snapshot aware, so it's going to search
> > everything constantly which is annoying.  I've got some time this
> > morning, I'll write up something different.  THanks,
>
> Ah, in that case I have around 20 snapshots, so yeah if it's going to be
> 20 times faster, I'll take that, especially if it ends up being one day
> instead of 3 weeks :)
>

Ok I pushed something new, but completely untested as I'm sitting at a
park with the kids and my kdevops thing is broken on my laptop.  You
should be able to do

btrfs rescue init-csum-tree <device>

and it'll rebuild the csum tree.  It'll give you a progress bar as
well.  I expect the normal amount of back and forth before it actually
works, but it should work faster for you.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-15 15:24                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
@ 2022-05-15 15:33                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  2022-05-15 15:35                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-15 15:33 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 15, 2022 at 11:24:34AM -0400, Josef Bacik wrote:
> Ok I pushed something new, but completely untested as I'm sitting at a
> park with the kids and my kdevops thing is broken on my laptop.  You
> should be able to do
> 
> btrfs rescue init-csum-tree <device>
> 
> and it'll rebuild the csum tree.  It'll give you a progress bar as
> well.  I expect the normal amount of back and forth before it actually
> works, but it should work faster for you.  Thanks,

Thanks. Actually I'm past that, I'm doing 
./btrfs check --init-csum-tree /dev/mapper/dshelf1
that's the one that's been running for days.

Are you saying init-csum-tree was moved to rescue which does run faster,
and after that I should run the last step, check --repair, that you
suggested?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-15 15:33                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
@ 2022-05-15 15:35                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  2022-05-15 15:41                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-15 15:35 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 15, 2022 at 11:33 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 15, 2022 at 11:24:34AM -0400, Josef Bacik wrote:
> > Ok I pushed something new, but completely untested as I'm sitting at a
> > park with the kids and my kdevops thing is broken on my laptop.  You
> > should be able to do
> >
> > btrfs rescue init-csum-tree <device>
> >
> > and it'll rebuild the csum tree.  It'll give you a progress bar as
> > well.  I expect the normal amount of back and forth before it actually
> > works, but it should work faster for you.  Thanks,
>
> Thanks. Actually I'm past that, I'm doing
> ./btrfs check --init-csum-tree /dev/mapper/dshelf1
> that's the one that's been running for days.
>
> Are you saying init-csum-tree was moved to rescue which does run faster,
> and after that I should run the last step, check --repair, that you
> suggested?
>

Yes, I've replaced the rescue --init-csum-tree with rescue
init-csum-tree, which will repopulate the csum extents.

So you should cancel what you're doing, and then run

btrfs rescue init-csum-tree <device>
btrfs check --repair <device>

and then you should be good to go.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-15 15:35                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
@ 2022-05-15 15:41                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  2022-05-15 15:48                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-15 15:41 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 15, 2022 at 11:35:28AM -0400, Josef Bacik wrote:
> On Sun, May 15, 2022 at 11:33 AM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sun, May 15, 2022 at 11:24:34AM -0400, Josef Bacik wrote:
> > > Ok I pushed something new, but completely untested as I'm sitting at a
> > > park with the kids and my kdevops thing is broken on my laptop.  You
> > > should be able to do
> > >
> > > btrfs rescue init-csum-tree <device>
> > >
> > > and it'll rebuild the csum tree.  It'll give you a progress bar as
> > > well.  I expect the normal amount of back and forth before it actually
> > > works, but it should work faster for you.  Thanks,
> >
> > Thanks. Actually I'm past that, I'm doing
> > ./btrfs check --init-csum-tree /dev/mapper/dshelf1
> > that's the one that's been running for days.
> >
> > Are you saying init-csum-tree was moved to rescue which does run faster,
> > and after that I should run the last step, check --repair, that you
> > suggested?
> >
> 
> Yes, I've replaced the rescue --init-csum-tree with rescue
> init-csum-tree, which will repopulate the csum extents.
> 
> So you should cancel what you're doing, and then run
> 
> btrfs rescue init-csum-tree <device>
> btrfs check --repair <device>
> 
> and then you should be good to go.  Thanks,

Gotcha, so let's work through the new version, looks like a recursive loop 

FS_INFO IS 0x55555564fbc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55555564fbc0

Program received signal SIGSEGV, Segmentation fault.
0x000055555559f121 in rb_search (root=root@entry=0x55555564ca20 <inserted>, key=key@entry=0x7fffff7ff020,
    comp=comp@entry=0x55555559af95 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffff7ff038)
    at common/rbtree-utils.c:54
54      {
(gdb) bt
#0  0x000055555559f121 in rb_search (root=root@entry=0x55555564ca20 <inserted>, key=key@entry=0x7fffff7ff020,
    comp=comp@entry=0x55555559af95 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffff7ff038)
    at common/rbtree-utils.c:54
#1  0x000055555559b199 in search_cache_extent (tree=tree@entry=0x55555564ca20 <inserted>,
    start=start@entry=15645196861440) at common/extent-cache.c:179
#2  0x0000555555584d27 in set_extent_bits (tree=tree@entry=0x55555564ca20 <inserted>, start=15645196861440,
    end=15645196877823, bits=bits@entry=1) at kernel-shared/extent_io.c:380
#3  0x0000555555584f5a in set_extent_dirty (tree=tree@entry=0x55555564ca20 <inserted>, start=<optimized out>,
    end=<optimized out>) at kernel-shared/extent_io.c:486
#4  0x00005555555e1c0e in record_csums_eb (eb=eb@entry=0x55555595fde0, processed=processed@entry=0x7fffffffdbe8)
    at cmds/rescue-init-csum-tree.c:203
#5  0x00005555555e1cf2 in record_csums_eb (eb=eb@entry=0x55555595fde0, processed=processed@entry=0x7fffffffdbe8)
    at cmds/rescue-init-csum-tree.c:224
#6  0x00005555555e1cf2 in record_csums_eb (eb=eb@entry=0x55555595fde0, processed=processed@entry=0x7fffffffdbe8)
    at cmds/rescue-init-csum-tree.c:224
#7  0x00005555555e1cf2 in record_csums_eb (eb=eb@entry=0x55555595fde0, processed=processed@entry=0x7fffffffdbe8)
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-15 15:41                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
@ 2022-05-15 15:48                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  2022-05-15 21:29                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-15 15:48 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 15, 2022 at 11:41 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 15, 2022 at 11:35:28AM -0400, Josef Bacik wrote:
> > On Sun, May 15, 2022 at 11:33 AM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sun, May 15, 2022 at 11:24:34AM -0400, Josef Bacik wrote:
> > > > Ok I pushed something new, but completely untested as I'm sitting at a
> > > > park with the kids and my kdevops thing is broken on my laptop.  You
> > > > should be able to do
> > > >
> > > > btrfs rescue init-csum-tree <device>
> > > >
> > > > and it'll rebuild the csum tree.  It'll give you a progress bar as
> > > > well.  I expect the normal amount of back and forth before it actually
> > > > works, but it should work faster for you.  Thanks,
> > >
> > > Thanks. Actually I'm past that, I'm doing
> > > ./btrfs check --init-csum-tree /dev/mapper/dshelf1
> > > that's the one that's been running for days.
> > >
> > > Are you saying init-csum-tree was moved to rescue which does run faster,
> > > and after that I should run the last step, check --repair, that you
> > > suggested?
> > >
> >
> > Yes, I've replaced the rescue --init-csum-tree with rescue
> > init-csum-tree, which will repopulate the csum extents.
> >
> > So you should cancel what you're doing, and then run
> >
> > btrfs rescue init-csum-tree <device>
> > btrfs check --repair <device>
> >
> > and then you should be good to go.  Thanks,
>
> Gotcha, so let's work through the new version, looks like a recursive loop
>

Fixed, thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-15 15:48                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
@ 2022-05-15 21:29                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  2022-05-15 23:01                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-15 21:29 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 15, 2022 at 11:48:00AM -0400, Josef Bacik wrote:
> Fixed, thanks,

Thanks, that was a whole lot quicker, less than 1h vs many days.


Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [76300, 108, 0]
misc/add0/file
doing an insert of the bytenr
doing an insert that overlaps our bytenr 10467695652864 8675328
processed 983040 of 108756992 possible bytes, 0%
Recording extents for root 165200
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165294
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165298
processed 524288 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
misc/add0/file
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165299
processed 16384 of 75792384 possible bytes, 0%
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes, 100%
doing block accounting
doing close???
Init extent tree finished, you can run check now

Progress! :)

Now check --repair will be running for while, will report back...

Deleting bad dir index [76854,96,27] root 163921
Deleting bad dir index [78134,96,26] root 163921
Deleting bad dir index [76854,96,28] root 163921
Deleting bad dir index [76854,96,30] root 163921
Deleting bad dir index [76854,96,32] root 163921
repairing missing dir index item for inode 86629
repairing missing dir index item for inode 86630
repairing missing dir index item for inode 86638
repairing missing dir index item for inode 86748
repairing missing dir index item for inode 86759
repairing missing dir index item for inode 86760
repairing missing dir index item for inode 86766
repairing missing dir index item for inode 87909
repairing missing dir index item for inode 87912
repairing missing dir index item for inode 87977
repairing missing dir index item for inode 87978
repairing missing dir index item for inode 87979
repairing missing dir index item for inode 87981
repairing missing dir index item for inode 87982
repairing missing dir index item for inode 87983
repairing missing dir index item for inode 87984
repairing missing dir index item for inode 87985
repairing missing dir index item for inode 87986
repairing missing dir index item for inode 87987
repairing missing dir index item for inode 87988
repairing missing dir index item for inode 87989
repairing missing dir index item for inode 95246
repairing missing dir index item for inode 95699
repairing missing dir index item for inode 95701
repairing missing dir index item for inode 95703
repairing missing dir index item for inode 78946
repairing missing dir index item for inode 78947
repairing missing dir index item for inode 78948
repairing missing dir index item for inode 78949
repairing missing dir index item for inode 78950
repairing missing dir index item for inode 78951
repairing missing dir index item for inode 78952
repairing missing dir index item for inode 78953
repairing missing dir index item for inode 78954
repairing missing dir index item for inode 78955
repairing missing dir index item for inode 78956
repairing missing dir index item for inode 78957
repairing missing dir index item for inode 78958
repairing missing dir index item for inode 78959
repairing missing dir index item for inode 78960
repairing missing dir index item for inode 78961
repairing missing dir index item for inode 78962
repairing missing dir index item for inode 78963
repairing missing dir index item for inode 78964
repairing missing dir index item for inode 78965
repairing missing dir index item for inode 78966
repairing missing dir index item for inode 78967
repairing missing dir index item for inode 78968
repairing missing dir index item for inode 78969
repairing missing dir index item for inode 78970
repairing missing dir index item for inode 78971
repairing missing dir index item for inode 78972
repairing missing dir index item for inode 78973
(...)

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-15 21:29                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
@ 2022-05-15 23:01                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-05-16  0:01                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-15 23:01 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 15, 2022 at 02:29:51PM -0700, Marc MERLIN wrote:
> Now check --repair will be running for while, will report back...
 
It wasn't as bad as I thought.
It fixed a bunch of things (long output, I have it saved if needed),
mostly orphanned stuff.
and finished with
root 165299 inode 95698 errors 1000, some csum missing
root 165299 inode 95699 errors 1000, some csum missing
root 165299 inode 95700 errors 1000, some csum missing
root 165299 inode 95701 errors 1000, some csum missing
root 165299 inode 95702 errors 1000, some csum missing
root 165299 inode 95703 errors 1000, some csum missing
root 165299 inode 95704 errors 1000, some csum missing
root 165299 inode 95705 errors 1000, some csum missing
ERROR: errors found in fs roots
found 56720129146880 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8334311424
total fs tree bytes: 7565082624
total extent tree bytes: 752779264
btree space waste bytes: 1336306596
file data blocks allocated: 59257396740096
 referenced 59313065607168

But I still can't mount the FS:
[1802750.985454] BTRFS info (device dm-1): disk space caching is enabled
[1802751.039629] BTRFS info (device dm-1): has skinny extents
[1802751.401992] BTRFS error (device dm-1): dev extent physical offset 941709328384 on devid 1 doesn't have corresponding chunk
[1802751.437568] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117
[1802751.482104] BTRFS error (device dm-1): open_ctree failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-15 23:01                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-05-16  0:01                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-05-16  0:57                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-16  0:01 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 15, 2022 at 7:01 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 15, 2022 at 02:29:51PM -0700, Marc MERLIN wrote:
> > Now check --repair will be running for while, will report back...
>
> It wasn't as bad as I thought.
> It fixed a bunch of things (long output, I have it saved if needed),
> mostly orphanned stuff.
> and finished with
> root 165299 inode 95698 errors 1000, some csum missing
> root 165299 inode 95699 errors 1000, some csum missing
> root 165299 inode 95700 errors 1000, some csum missing
> root 165299 inode 95701 errors 1000, some csum missing
> root 165299 inode 95702 errors 1000, some csum missing
> root 165299 inode 95703 errors 1000, some csum missing
> root 165299 inode 95704 errors 1000, some csum missing
> root 165299 inode 95705 errors 1000, some csum missing
> ERROR: errors found in fs roots
> found 56720129146880 bytes used, error(s) found
> total csum bytes: 0
> total tree bytes: 8334311424
> total fs tree bytes: 7565082624
> total extent tree bytes: 752779264
> btree space waste bytes: 1336306596
> file data blocks allocated: 59257396740096
>  referenced 59313065607168
>
> But I still can't mount the FS:
> [1802750.985454] BTRFS info (device dm-1): disk space caching is enabled
> [1802751.039629] BTRFS info (device dm-1): has skinny extents
> [1802751.401992] BTRFS error (device dm-1): dev extent physical offset 941709328384 on devid 1 doesn't have corresponding chunk
> [1802751.437568] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117
> [1802751.482104] BTRFS error (device dm-1): open_ctree failed
>

Huh, did this pop during the btrfs check?  We can just delete that guy

btrfs-corrupt-block -d "1,204,941709328384" -r 3 <device>

and then you should be good, unless there are other dangling dev
extents that need to be removed.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-16  0:01                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-05-16  0:57                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-05-16 14:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-16  0:57 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 15, 2022 at 08:01:02PM -0400, Josef Bacik wrote:
> Huh, did this pop during the btrfs check?  We can just delete that guy
 
Yes, just at the end, when it looked finished

> btrfs-corrupt-block -d "1,204,941709328384" -r 3 <device>
> 
> and then you should be good, unless there are other dangling dev
> extents that need to be removed.  Thanks,

Is that bad?
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1,204,941709328384" -r 3 /dev/mapper/dshelf1
FS_INFO IS 0x56044c4bd600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x56044c4bd600
Error searching to node -2

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-16  0:57                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-05-16 14:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-05-16 15:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-16 14:50 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 15, 2022 at 8:58 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 15, 2022 at 08:01:02PM -0400, Josef Bacik wrote:
> > Huh, did this pop during the btrfs check?  We can just delete that guy
>
> Yes, just at the end, when it looked finished
>
> > btrfs-corrupt-block -d "1,204,941709328384" -r 3 <device>
> >
> > and then you should be good, unless there are other dangling dev
> > extents that need to be removed.  Thanks,
>
> Is that bad?

Yeah, means I don't understand my own filesystem, use -r 4 instead
please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-16 14:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-05-16 15:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-05-16 15:31                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-05-16 15:36                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 2 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-05-16 15:16 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, May 16, 2022 at 10:50:42AM -0400, Josef Bacik wrote:
> > > btrfs-corrupt-block -d "1,204,941709328384" -r 3 <device>
> > >
> > > and then you should be good, unless there are other dangling dev
> > > extents that need to be removed.  Thanks,
> >
> > Is that bad?
> 
> Yeah, means I don't understand my own filesystem, use -r 4 instead
> please.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1,204,941709328384" -r4 /dev/mapper/dshelf1 
FS_INFO IS 0x55e58be9e600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55e58be9e600

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1,204,941709328384" -r 4 /dev/mapper/dshelf1 
FS_INFO IS 0x55e239055600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55e239055600
Error searching to node -2

Means it worked?
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-16 15:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-05-16 15:31                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-05-16 15:36                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  1 sibling, 0 replies; 479+ messages in thread
From: Josef Bacik @ 2022-05-16 15:31 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 16, 2022 at 11:16 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 16, 2022 at 10:50:42AM -0400, Josef Bacik wrote:
> > > > btrfs-corrupt-block -d "1,204,941709328384" -r 3 <device>
> > > >
> > > > and then you should be good, unless there are other dangling dev
> > > > extents that need to be removed.  Thanks,
> > >
> > > Is that bad?
> >
> > Yeah, means I don't understand my own filesystem, use -r 4 instead
> > please.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1,204,941709328384" -r4 /dev/mapper/dshelf1
> FS_INFO IS 0x55e58be9e600
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x55e58be9e600
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1,204,941709328384" -r 4 /dev/mapper/dshelf1
> FS_INFO IS 0x55e239055600
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x55e239055600
> Error searching to node -2
>
> Means it worked?

Yeah, probably should print out a "Error: Success!" when it does
actually do something.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-16 15:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-05-16 15:31                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-05-16 15:36                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  2022-05-16 16:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-16 15:36 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, May 16, 2022 at 08:16:53AM -0700, Marc MERLIN wrote:
> On Mon, May 16, 2022 at 10:50:42AM -0400, Josef Bacik wrote:
> > > > btrfs-corrupt-block -d "1,204,941709328384" -r 3 <device>
> > > >
> > > > and then you should be good, unless there are other dangling dev
> > > > extents that need to be removed.  Thanks,
> > >
> > > Is that bad?
> > 
> > Yeah, means I don't understand my own filesystem, use -r 4 instead
> > please.  Thanks,
> 
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1,204,941709328384" -r4 /dev/mapper/dshelf1 
> FS_INFO IS 0x55e58be9e600
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x55e58be9e600
> 
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1,204,941709328384" -r 4 /dev/mapper/dshelf1 
> FS_INFO IS 0x55e239055600
> JOSEF: root 9
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x55e239055600
> Error searching to node -2
> 
> Means it worked?

Could you update the code so that it says "deleted"?

I'm doing repair again, and I now see:
Fixed 0 roots.
[2/7] checking extents
checksum verify failed on 364637798400 wanted 0xc3ba5144 found 0x30a5fff1
checksum verify failed on 364637962240 wanted 0x91d33e8a found 0x00890be7
checksum verify failed on 11970891186176 wanted 0x6b46adfb found 0xe40e4cbe
checksum verify failed on 12511729680384 wanted 0x41be9fa6 found 0x01f06e9c
checksum verify failed on 12512010420224 wanted 0x365027e7 found 0xba73f602
checksum verify failed on 12512303677440 wanted 0xc08fc7a0 found 0x3cb30242
checksum verify failed on 13576951021568 wanted 0x9511e976 found 0xd8fe35ea
checksum verify failed on 13577013624832 wanted 0x76bbe220 found 0xdfabbbdb
checksum verify failed on 13577013919744 wanted 0xafb37267 found 0x69504e0d
checksum verify failed on 13577113485312 wanted 0x049c1e5a found 0x0ddcba27
checksum verify failed on 13577170976768 wanted 0x9585372b found 0x048e24e3
checksum verify failed on 13577420161024 wanted 0x6d37d200 found 0x45f7a0b8
checksum verify failed on 13577490890752 wanted 0x426c3628 found 0x35008768
checksum verify failed on 13577726541824 wanted 0xc3089b25 found 0x426c57c7
checksum verify failed on 13577776611328 wanted 0x50eba933 found 0x4f0ec122
checksum verify failed on 364983844864 wanted 0x3a556bff found 0xba7431e0
checksum verify failed on 364986433536 wanted 0xa2ee2364 found 0xc0023b14
checksum verify failed on 364986449920 wanted 0x55bf9ee9 found 0x9cda5c9f
checksum verify failed on 364986482688 wanted 0x5421a4b7 found 0x37ac4734
(...)
checksum verify failed on 15646006722560 wanted 0x760a06a0 found 0xb44fcd95
checksum verify failed on 15645319872512 wanted 0x3bfb10c8 found 0x2be82f60
checksum verify failed on 15645980491776 wanted 0xe1e674a7 found 0x0fdfda2c
checksum verify failed on 15645526884352 wanted 0xc2d409c1 found 0xb3eee8d2
checksum verify failed on 13577013936128 wanted 0x4ba00b03 found 0x64614751
Chunk[256, 228, 4523166793728] stripe[1, 4531228246016] is not found in dev extent
Chunk[256, 228, 4524240535552] stripe[1, 4532301987840] is not found in dev extent
Chunk[256, 228, 4525314277376] stripe[1, 4533375729664] is not found in dev extent
Chunk[256, 228, 4526388019200] stripe[1, 4534449471488] is not found in dev extent
Chunk[256, 228, 4527461761024] stripe[1, 4535523213312] is not found in dev extent
Chunk[256, 228, 4528535502848] stripe[1, 4536596955136] is not found in dev extent
Chunk[256, 228, 4529609244672] stripe[1, 4537670696960] is not found in dev extent
(...)
Chunk[256, 228, 15362624061440] stripe[1, 14782241439744] is not found in dev extent
Chunk[256, 228, 15363697803264] stripe[1, 14783315181568] is not found in dev extent
Chunk[256, 228, 15364771545088] stripe[1, 14784388923392] is not found in dev extent
Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent
Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent
Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk.
Device extent[1, 11595375706112, 1073741824] didn't find the relative chunk.
Device extent[1, 11596449447936, 1073741824] didn't find the relative chunk.
Device extent[1, 11597523189760, 1073741824] didn't find the relative chunk.
Device extent[1, 11598596931584, 1073741824] didn't find the relative chunk.
Device extent[1, 11599670673408, 1073741824] didn't find the relative chunk.
Device extent[1, 11600744415232, 1073741824] didn't find the relative chunk.
Device extent[1, 11601818157056, 1073741824] didn't find the relative chunk.
Device extent[1, 11611481833472, 1073741824] didn't find the relative chunk.
Device extent[1, 11612555575296, 1073741824] didn't find the relative chunk.
Device extent[1, 11613629317120, 1073741824] didn't find the relative chunk.
(...)
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
super bytes used 14180042272768 mismatches actual used 14180042223616


Ok to continue?
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-16 15:36                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
@ 2022-05-16 16:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-05-16 16:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-16 16:53 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, May 16, 2022 at 08:36:51AM -0700, Marc MERLIN wrote:
> Could you update the code so that it says "deleted"?
 
AI on this one? ^^^

> I'm doing repair again, and I now see:
> Fixed 0 roots.
> [2/7] checking extents
> checksum verify failed on 364637798400 wanted 0xc3ba5144 found 0x30a5fff1
> checksum verify failed on 364637962240 wanted 0x91d33e8a found 0x00890be7
> checksum verify failed on 11970891186176 wanted 0x6b46adfb found 0xe40e4cbe
> checksum verify failed on 12511729680384 wanted 0x41be9fa6 found 0x01f06e9c
> checksum verify failed on 12512010420224 wanted 0x365027e7 found 0xba73f602
> checksum verify failed on 12512303677440 wanted 0xc08fc7a0 found 0x3cb30242
> checksum verify failed on 13576951021568 wanted 0x9511e976 found 0xd8fe35ea
> checksum verify failed on 13577013624832 wanted 0x76bbe220 found 0xdfabbbdb
> checksum verify failed on 13577013919744 wanted 0xafb37267 found 0x69504e0d
> checksum verify failed on 13577113485312 wanted 0x049c1e5a found 0x0ddcba27
> checksum verify failed on 13577170976768 wanted 0x9585372b found 0x048e24e3
> checksum verify failed on 13577420161024 wanted 0x6d37d200 found 0x45f7a0b8
> checksum verify failed on 13577490890752 wanted 0x426c3628 found 0x35008768
> checksum verify failed on 13577726541824 wanted 0xc3089b25 found 0x426c57c7
> checksum verify failed on 13577776611328 wanted 0x50eba933 found 0x4f0ec122
> checksum verify failed on 364983844864 wanted 0x3a556bff found 0xba7431e0
> checksum verify failed on 364986433536 wanted 0xa2ee2364 found 0xc0023b14
> checksum verify failed on 364986449920 wanted 0x55bf9ee9 found 0x9cda5c9f
> checksum verify failed on 364986482688 wanted 0x5421a4b7 found 0x37ac4734
> (...)
> checksum verify failed on 15646006722560 wanted 0x760a06a0 found 0xb44fcd95
> checksum verify failed on 15645319872512 wanted 0x3bfb10c8 found 0x2be82f60
> checksum verify failed on 15645980491776 wanted 0xe1e674a7 found 0x0fdfda2c
> checksum verify failed on 15645526884352 wanted 0xc2d409c1 found 0xb3eee8d2
> checksum verify failed on 13577013936128 wanted 0x4ba00b03 found 0x64614751
> Chunk[256, 228, 4523166793728] stripe[1, 4531228246016] is not found in dev extent
> Chunk[256, 228, 4524240535552] stripe[1, 4532301987840] is not found in dev extent
> Chunk[256, 228, 4525314277376] stripe[1, 4533375729664] is not found in dev extent
> Chunk[256, 228, 4526388019200] stripe[1, 4534449471488] is not found in dev extent
> Chunk[256, 228, 4527461761024] stripe[1, 4535523213312] is not found in dev extent
> Chunk[256, 228, 4528535502848] stripe[1, 4536596955136] is not found in dev extent
> Chunk[256, 228, 4529609244672] stripe[1, 4537670696960] is not found in dev extent
> (...)
> Chunk[256, 228, 15362624061440] stripe[1, 14782241439744] is not found in dev extent
> Chunk[256, 228, 15363697803264] stripe[1, 14783315181568] is not found in dev extent
> Chunk[256, 228, 15364771545088] stripe[1, 14784388923392] is not found in dev extent
> Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent
> Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent
> Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk.
> Device extent[1, 11595375706112, 1073741824] didn't find the relative chunk.
> Device extent[1, 11596449447936, 1073741824] didn't find the relative chunk.
> Device extent[1, 11597523189760, 1073741824] didn't find the relative chunk.
> Device extent[1, 11598596931584, 1073741824] didn't find the relative chunk.
> Device extent[1, 11599670673408, 1073741824] didn't find the relative chunk.
> Device extent[1, 11600744415232, 1073741824] didn't find the relative chunk.
> Device extent[1, 11601818157056, 1073741824] didn't find the relative chunk.
> Device extent[1, 11611481833472, 1073741824] didn't find the relative chunk.
> Device extent[1, 11612555575296, 1073741824] didn't find the relative chunk.
> Device extent[1, 11613629317120, 1073741824] didn't find the relative chunk.
> (...)
> Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
> Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
> super bytes used 14180042272768 mismatches actual used 14180042223616
> 
> Ok to continue?

It didn't take that long, so I let it finish.

root 165299 inode 95699 errors 1000, some csum missing
root 165299 inode 95700 errors 1000, some csum missing
root 165299 inode 95701 errors 1000, some csum missing
root 165299 inode 95702 errors 1000, some csum missing
root 165299 inode 95703 errors 1000, some csum missing
root 165299 inode 95704 errors 1000, some csum missing
root 165299 inode 95705 errors 1000, some csum missing
ERROR: errors found in fs roots
Fixed the nlink of inode 74607
reset isize for dir 74754 root 165098
reset isize for dir 74978 root 165098
reset isize for dir 75036 root 165098
reset isize for dir 75038 root 165098
reset isize for dir 76276 root 165098
reset isize for dir 74754 root 165198
reset isize for dir 74978 root 165198
reset isize for dir 75036 root 165198
reset isize for dir 75038 root 165198
reset isize for dir 75937 root 165198
reset isize for dir 76276 root 165198
Fixed the nlink of inode 76565
reset isize for dir 74754 root 165298
reset isize for dir 74978 root 165298
reset isize for dir 75036 root 165298
reset isize for dir 75038 root 165298
reset isize for dir 76276 root 165298
found 226880675774464 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 33496432640
total fs tree bytes: 30395858944
total extent tree bytes: 3034775552
btree space waste bytes: 5386252957
file data blocks allocated: 249664874872832
 referenced 249887841910784
gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro,recovery /dev/mapper/dshelf1 /mnt/mnt
mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error.

BTRFS info (device dm-1): flagging fs with big metadata feature
BTRFS warning (device dm-1): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
BTRFS info (device dm-1): trying to use backup root at mount time
BTRFS info (device dm-1): disk space caching is enabled
BTRFS info (device dm-1): has skinny extents
BTRFS error (device dm-1): dev extent physical offset 943856812032 on devid 1 doesn't have corresponding chunk
BTRFS error (device dm-1): failed to verify dev extents against chunks: -117
BTRFS error (device dm-1): open_ctree failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-16 16:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-05-16 16:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-05-17 19:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-16 16:55 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 16, 2022 at 12:53 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 16, 2022 at 08:36:51AM -0700, Marc MERLIN wrote:
> > Could you update the code so that it says "deleted"?
>
> AI on this one? ^^^
>
> > I'm doing repair again, and I now see:
> > Fixed 0 roots.
> > [2/7] checking extents
> > checksum verify failed on 364637798400 wanted 0xc3ba5144 found 0x30a5fff1
> > checksum verify failed on 364637962240 wanted 0x91d33e8a found 0x00890be7
> > checksum verify failed on 11970891186176 wanted 0x6b46adfb found 0xe40e4cbe
> > checksum verify failed on 12511729680384 wanted 0x41be9fa6 found 0x01f06e9c
> > checksum verify failed on 12512010420224 wanted 0x365027e7 found 0xba73f602
> > checksum verify failed on 12512303677440 wanted 0xc08fc7a0 found 0x3cb30242
> > checksum verify failed on 13576951021568 wanted 0x9511e976 found 0xd8fe35ea
> > checksum verify failed on 13577013624832 wanted 0x76bbe220 found 0xdfabbbdb
> > checksum verify failed on 13577013919744 wanted 0xafb37267 found 0x69504e0d
> > checksum verify failed on 13577113485312 wanted 0x049c1e5a found 0x0ddcba27
> > checksum verify failed on 13577170976768 wanted 0x9585372b found 0x048e24e3
> > checksum verify failed on 13577420161024 wanted 0x6d37d200 found 0x45f7a0b8
> > checksum verify failed on 13577490890752 wanted 0x426c3628 found 0x35008768
> > checksum verify failed on 13577726541824 wanted 0xc3089b25 found 0x426c57c7
> > checksum verify failed on 13577776611328 wanted 0x50eba933 found 0x4f0ec122
> > checksum verify failed on 364983844864 wanted 0x3a556bff found 0xba7431e0
> > checksum verify failed on 364986433536 wanted 0xa2ee2364 found 0xc0023b14
> > checksum verify failed on 364986449920 wanted 0x55bf9ee9 found 0x9cda5c9f
> > checksum verify failed on 364986482688 wanted 0x5421a4b7 found 0x37ac4734
> > (...)
> > checksum verify failed on 15646006722560 wanted 0x760a06a0 found 0xb44fcd95
> > checksum verify failed on 15645319872512 wanted 0x3bfb10c8 found 0x2be82f60
> > checksum verify failed on 15645980491776 wanted 0xe1e674a7 found 0x0fdfda2c
> > checksum verify failed on 15645526884352 wanted 0xc2d409c1 found 0xb3eee8d2
> > checksum verify failed on 13577013936128 wanted 0x4ba00b03 found 0x64614751
> > Chunk[256, 228, 4523166793728] stripe[1, 4531228246016] is not found in dev extent
> > Chunk[256, 228, 4524240535552] stripe[1, 4532301987840] is not found in dev extent
> > Chunk[256, 228, 4525314277376] stripe[1, 4533375729664] is not found in dev extent
> > Chunk[256, 228, 4526388019200] stripe[1, 4534449471488] is not found in dev extent
> > Chunk[256, 228, 4527461761024] stripe[1, 4535523213312] is not found in dev extent
> > Chunk[256, 228, 4528535502848] stripe[1, 4536596955136] is not found in dev extent
> > Chunk[256, 228, 4529609244672] stripe[1, 4537670696960] is not found in dev extent
> > (...)
> > Chunk[256, 228, 15362624061440] stripe[1, 14782241439744] is not found in dev extent
> > Chunk[256, 228, 15363697803264] stripe[1, 14783315181568] is not found in dev extent
> > Chunk[256, 228, 15364771545088] stripe[1, 14784388923392] is not found in dev extent
> > Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent
> > Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent
> > Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk.
> > Device extent[1, 11595375706112, 1073741824] didn't find the relative chunk.
> > Device extent[1, 11596449447936, 1073741824] didn't find the relative chunk.
> > Device extent[1, 11597523189760, 1073741824] didn't find the relative chunk.
> > Device extent[1, 11598596931584, 1073741824] didn't find the relative chunk.
> > Device extent[1, 11599670673408, 1073741824] didn't find the relative chunk.
> > Device extent[1, 11600744415232, 1073741824] didn't find the relative chunk.
> > Device extent[1, 11601818157056, 1073741824] didn't find the relative chunk.
> > Device extent[1, 11611481833472, 1073741824] didn't find the relative chunk.
> > Device extent[1, 11612555575296, 1073741824] didn't find the relative chunk.
> > Device extent[1, 11613629317120, 1073741824] didn't find the relative chunk.
> > (...)
> > Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
> > Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.

I'm going to have to fix check to fix these problems, I'll let you
know when I have something.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-16 16:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-05-17 19:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  2022-05-17 20:27                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-17 19:49 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 16, 2022 at 12:55 PM Josef Bacik <josef@toxicpanda.com> wrote:
>
> On Mon, May 16, 2022 at 12:53 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Mon, May 16, 2022 at 08:36:51AM -0700, Marc MERLIN wrote:
> > > Could you update the code so that it says "deleted"?
> >
> > AI on this one? ^^^
> >
> > > I'm doing repair again, and I now see:
> > > Fixed 0 roots.
> > > [2/7] checking extents
> > > checksum verify failed on 364637798400 wanted 0xc3ba5144 found 0x30a5fff1
> > > checksum verify failed on 364637962240 wanted 0x91d33e8a found 0x00890be7
> > > checksum verify failed on 11970891186176 wanted 0x6b46adfb found 0xe40e4cbe
> > > checksum verify failed on 12511729680384 wanted 0x41be9fa6 found 0x01f06e9c
> > > checksum verify failed on 12512010420224 wanted 0x365027e7 found 0xba73f602
> > > checksum verify failed on 12512303677440 wanted 0xc08fc7a0 found 0x3cb30242
> > > checksum verify failed on 13576951021568 wanted 0x9511e976 found 0xd8fe35ea
> > > checksum verify failed on 13577013624832 wanted 0x76bbe220 found 0xdfabbbdb
> > > checksum verify failed on 13577013919744 wanted 0xafb37267 found 0x69504e0d
> > > checksum verify failed on 13577113485312 wanted 0x049c1e5a found 0x0ddcba27
> > > checksum verify failed on 13577170976768 wanted 0x9585372b found 0x048e24e3
> > > checksum verify failed on 13577420161024 wanted 0x6d37d200 found 0x45f7a0b8
> > > checksum verify failed on 13577490890752 wanted 0x426c3628 found 0x35008768
> > > checksum verify failed on 13577726541824 wanted 0xc3089b25 found 0x426c57c7
> > > checksum verify failed on 13577776611328 wanted 0x50eba933 found 0x4f0ec122
> > > checksum verify failed on 364983844864 wanted 0x3a556bff found 0xba7431e0
> > > checksum verify failed on 364986433536 wanted 0xa2ee2364 found 0xc0023b14
> > > checksum verify failed on 364986449920 wanted 0x55bf9ee9 found 0x9cda5c9f
> > > checksum verify failed on 364986482688 wanted 0x5421a4b7 found 0x37ac4734
> > > (...)
> > > checksum verify failed on 15646006722560 wanted 0x760a06a0 found 0xb44fcd95
> > > checksum verify failed on 15645319872512 wanted 0x3bfb10c8 found 0x2be82f60
> > > checksum verify failed on 15645980491776 wanted 0xe1e674a7 found 0x0fdfda2c
> > > checksum verify failed on 15645526884352 wanted 0xc2d409c1 found 0xb3eee8d2
> > > checksum verify failed on 13577013936128 wanted 0x4ba00b03 found 0x64614751
> > > Chunk[256, 228, 4523166793728] stripe[1, 4531228246016] is not found in dev extent
> > > Chunk[256, 228, 4524240535552] stripe[1, 4532301987840] is not found in dev extent
> > > Chunk[256, 228, 4525314277376] stripe[1, 4533375729664] is not found in dev extent
> > > Chunk[256, 228, 4526388019200] stripe[1, 4534449471488] is not found in dev extent
> > > Chunk[256, 228, 4527461761024] stripe[1, 4535523213312] is not found in dev extent
> > > Chunk[256, 228, 4528535502848] stripe[1, 4536596955136] is not found in dev extent
> > > Chunk[256, 228, 4529609244672] stripe[1, 4537670696960] is not found in dev extent
> > > (...)
> > > Chunk[256, 228, 15362624061440] stripe[1, 14782241439744] is not found in dev extent
> > > Chunk[256, 228, 15363697803264] stripe[1, 14783315181568] is not found in dev extent
> > > Chunk[256, 228, 15364771545088] stripe[1, 14784388923392] is not found in dev extent
> > > Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent
> > > Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent
> > > Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk.
> > > Device extent[1, 11595375706112, 1073741824] didn't find the relative chunk.
> > > Device extent[1, 11596449447936, 1073741824] didn't find the relative chunk.
> > > Device extent[1, 11597523189760, 1073741824] didn't find the relative chunk.
> > > Device extent[1, 11598596931584, 1073741824] didn't find the relative chunk.
> > > Device extent[1, 11599670673408, 1073741824] didn't find the relative chunk.
> > > Device extent[1, 11600744415232, 1073741824] didn't find the relative chunk.
> > > Device extent[1, 11601818157056, 1073741824] didn't find the relative chunk.
> > > Device extent[1, 11611481833472, 1073741824] didn't find the relative chunk.
> > > Device extent[1, 11612555575296, 1073741824] didn't find the relative chunk.
> > > Device extent[1, 11613629317120, 1073741824] didn't find the relative chunk.
> > > (...)
> > > Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
> > > Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
>
> I'm going to have to fix check to fix these problems, I'll let you
> know when I have something.  Thanks,
>

Sorry, kids coming down with COVID.  I've pushed a fix for check to
delete these things, you can run btrfs check --repair and it should do
the right thing.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-17 19:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
@ 2022-05-17 20:27                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
  2022-05-17 20:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-17 20:27 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, May 17, 2022 at 03:49:50PM -0400, Josef Bacik wrote:
> Sorry, kids coming down with COVID.  I've pushed a fix for check to

Oh no, sorry to hear. Hopefully they'll be ok.

> delete these things, you can run btrfs check --repair and it should do
> the right thing.  Thanks,

Thanks for not giving up. I'm really surprised by the amount of damage that this filesystem
endured in less than 60 seconds, especially when I'm pretty sure I wasn't even writing to it.
Although I understand that some extra damage may have been done during the repair attempts.


(...)
checksum verify failed on 11160662736896 wanted 0xc46d6447 found 0x9e6e915e
checksum verify failed on 11651750428672 wanted 0x1a5d3be2 found 0xb0ee91ed
checksum verify failed on 11652259332096 wanted 0x779cd76b found 0xa10261c1
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
checksum verify failed on 15645125918720 wanted 0xa2327e31 found 0xe27a2894
checksum verify failed on 15645131276288 wanted 0x578d8fc7 found 0xc159231d
checksum verify failed on 15645146775552 wanted 0x57d7eafa found 0xac917df2
(...)
Chunk[256, 228, 15357255352320] stripe[1, 14776872730624] is not found in dev extent
Chunk[256, 228, 15358329094144] stripe[1, 14777946472448] is not found in dev extent
Chunk[256, 228, 15359402835968] stripe[1, 14779020214272] is not found in dev extent
Chunk[256, 228, 15360476577792] stripe[1, 14780093956096] is not found in dev extent
Chunk[256, 228, 15361550319616] stripe[1, 14781167697920] is not found in dev extent
Chunk[256, 228, 15362624061440] stripe[1, 14782241439744] is not found in dev extent
Chunk[256, 228, 15363697803264] stripe[1, 14783315181568] is not found in dev extent
Chunk[256, 228, 15364771545088] stripe[1, 14784388923392] is not found in dev extent
Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent
Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent
Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk.
ERROR: couldn't delete chunk record
incorrect local backref count on 2952871936 parent 13576823652352 owner 0 offset 0 found 0 wanted 1 back 0x55aca7a699d0
backref disk bytenr does not match extent record, bytenr=2952871936, ref bytenr=0
data backref 2952871936 root 11223 owner 258 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 2952871936 root 11223 owner 258 offset 0 found 1 wanted 0 back 0x55acc87dc040
backpointer mismatch on [2952871936 262144]
ERROR: attempt to start transaction over already running one
failed to repair damaged filesystem, aborting

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-17 20:27                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
@ 2022-05-17 20:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  2022-05-17 21:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-17 20:39 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, May 17, 2022 at 4:27 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, May 17, 2022 at 03:49:50PM -0400, Josef Bacik wrote:
> > Sorry, kids coming down with COVID.  I've pushed a fix for check to
>
> Oh no, sorry to hear. Hopefully they'll be ok.
>
> > delete these things, you can run btrfs check --repair and it should do
> > the right thing.  Thanks,
>
> Thanks for not giving up. I'm really surprised by the amount of damage that this filesystem
> endured in less than 60 seconds, especially when I'm pretty sure I wasn't even writing to it.
> Although I understand that some extra damage may have been done during the repair attempts.
>

I'm an idiot, wrong root for the dev extents.

What I *think* happened is you essentially lost random blocks from 1
transaction.  This isn't particularly harmful, except you had a chunk
allocation that happened, so you lost some of the chunk root.  I
didn't realize this is what was happening until now.  If I had I
probably could have pieced together these dev extents into chunks, and
then all those blocks that we had that didn't map to a chunk probably
would have been fine, because they were in the range that we lost.

It's not that you lost a lot of blocks, you just lost a few really,
really important blocks, and then I failed to recognize what happened
so I threw out more than I should have, plus all the other little
wonkiness from experimental recovery tools.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-17 20:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
@ 2022-05-17 21:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  2022-05-18 18:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-17 21:22 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, May 17, 2022 at 04:39:18PM -0400, Josef Bacik wrote:
> What I *think* happened is you essentially lost random blocks from 1
> transaction.  This isn't particularly harmful, except you had a chunk
> allocation that happened, so you lost some of the chunk root.  I
> didn't realize this is what was happening until now.  If I had I
> probably could have pieced together these dev extents into chunks, and
> then all those blocks that we had that didn't map to a chunk probably
> would have been fine, because they were in the range that we lost.
> 
> It's not that you lost a lot of blocks, you just lost a few really,
> really important blocks, and then I failed to recognize what happened
> so I threw out more than I should have, plus all the other little
> wonkiness from experimental recovery tools.  Thanks,

Right, which is totally understandable. I'm still glad I have backups
(waiting to restore), but I very much appreciate your work right now as
it's fixing the bigger problem of recovering an unmountable filesystem.

Ultimately I don't care as much about how much data is lost, but I very
much care about btrfs check repair getting the filesystem back to a
mountable state, so I've been more than happy helping you get to that
state and I'm very thankful for all the time you spent on this so far.
Hopefully we're almost there :)

For extra points, files that have corrupted checksums, I'd love for them
to be renamed to file.corrupted, so that someone who cares about partial
data, can get some data back, and that you can recover much faster with
rsync if it only has to look for missing files, as opposed to running a
checksum on all the data on both sides, which would take ages.
I know in this case, I may end up with some amount of files silently
corrupted since we rebuilt the checksum table, not sure if there is a
way to do this like I described or some similar way, in the future.

I see you pushed another change, trying it now.

I have the full output saved if need be.

Device extent[1, 14812306210816, 1073741824] didn't find the relative chunk.
Device extent[1, 9544528822272, 1073741824] didn't find the relative chunk.
Device extent[1, 11231377227776, 1073741824] didn't find the relative chunk.
Device extent[1, 11406397145088, 1073741824] didn't find the relative chunk.
Device extent[1, 11416060821504, 1073741824] didn't find the relative chunk.
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
incorrect local backref count on 2952871936 parent 13576823652352 owner 0 offset 0 found 0 wanted 1 back 0x55db601ea230
backref disk bytenr does not match extent record, bytenr=2952871936, ref bytenr=0
data backref 2952871936 root 11223 owner 258 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 2952871936 root 11223 owner 258 offset 0 found 1 wanted 0 back 0x55db80f5c6a0
backpointer mismatch on [2952871936 262144]
repair deleting extent record: key [2952871936,168,262144]
adding new data backref on 2952871936 root 1 owner 258 offset 0 found 1
adding new data backref on 2952871936 root 11223 owner 258 offset 0 found 1
Repaired extent references for 2952871936
incorrect local backref count on 4156227584 parent 13576823652352 owner 0 offset 0 found 0 wanted 1 back 0x55db607ad1f0
backref disk bytenr does not match extent record, bytenr=4156227584, ref bytenr=0
data backref 4156227584 root 11223 owner 259 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 4156227584 root 11223 owner 259 offset 0 found 1 wanted 0 back 0x55db80f5c570
backpointer mismatch on [4156227584 262144]
repair deleting extent record: key [4156227584,168,262144]
adding new data backref on 4156227584 root 1 owner 259 offset 0 found 1
adding new data backref on 4156227584 root 11223 owner 259 offset 0 found 1
Repaired extent references for 4156227584
(...)
adding new data backref on 10449820549120 root 1 owner 256 offset 0 found 1
adding new data backref on 10449820549120 root 11223 owner 256 offset 0 found 1
Repaired extent references for 10449820549120
super bytes used 14180042240000 mismatches actual used 14180042256384
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Block group[4523166793728, 1073741824] (flags = 1) didn't find the relative chunk.
Block group[4524240535552, 1073741824] (flags = 1) didn't find the relative chunk.
Block group[4525314277376, 1073741824] (flags = 1) didn't find the relative chunk.
Block group[4526388019200, 1073741824] (flags = 1) didn't find the relative chunk.
Block group[4527461761024, 1073741824] (flags = 1) didn't find the relative chunk.
(...)
super bytes used 14180042240000 mismatches actual used 14180042272768
(...)
root 159785 inode 74938 errors 1000, some csum missing
root 159785 inode 74945 errors 1000, some csum missing
root 159785 inode 74952 errors 1100, file extent discount, some csum missing
Found file extent holes:
        start: 2477785088, len: 934416384
root 159785 inode 74958 errors 1500, file extent discount, nbytes wrong, some csum missing
Found file extent holes:
        start: 49020928, len: 1426894848
root 159785 inode 74964 errors 1100, file extent discount, some csum missing
Found file extent holes:
        start: 714735616, len: 24973312
root 159785 inode 74966 errors 1500, file extent discount, nbytes wrong, some csum missing
Found file extent holes:
        start: 13893632, len: 838578176
root 159785 inode 74967 errors 1100, file extent discount, some csum missing
Found file extent holes:
        start: 86638592, len: 357191680
root 159785 inode 74977 errors 1000, some csum missing
root 159785 inode 74983 errors 1000, some csum missing
root 159785 inode 74984 errors 1000, some csum missing
(...)
root 159785 inode 75025 errors 1000, some csum missing
root 159785 inode 75034 errors 1500, file extent discount, nbytes wrong, some csum missing
Found file extent holes:
        start: 132841472, len: 112558080
(...)
root 165299 inode 95692 errors 1000, some csum missing
root 165299 inode 95697 errors 1000, some csum missing
root 165299 inode 95698 errors 1000, some csum missing
root 165299 inode 95699 errors 1000, some csum missing
root 165299 inode 95700 errors 1000, some csum missing
root 165299 inode 95701 errors 1000, some csum missing
root 165299 inode 95702 errors 1000, some csum missing
root 165299 inode 95703 errors 1000, some csum missing
root 165299 inode 95704 errors 1000, some csum missing
root 165299 inode 95705 errors 1000, some csum missing
ERROR: errors found in fs roots

Starting repair.
Opening filesystem to check...
JOSEF: root 9
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
reset devid 1 bytes_used to 14599692746752
No device size related problem found
cache and super generation don't match, space cache will be invalidated
found 85080253587456 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 12561334272
total fs tree bytes: 11398840320
total extent tree bytes: 1138114560
btree space waste bytes: 2020110583
file data blocks allocated: 93624650268672
 referenced 93708263153664


[1969680.027665] BTRFS info (device dm-1): flagging fs with big metadata feature
[1969680.049987] BTRFS warning (device dm-1): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
[1969680.049991] BTRFS info (device dm-1): trying to use backup root at mount time
[1969680.049992] BTRFS info (device dm-1): disk space caching is enabled
[1969680.049993] BTRFS info (device dm-1): has skinny extents
[1969681.639536] BTRFS error (device dm-1): logical 4523166793728 len 1073741824 found bg but no related chunk
[1969681.670760] BTRFS error (device dm-1): failed to read block groups: -2
[1969681.703366] BTRFS error (device dm-1): open_ctree failed




Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-17 21:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
@ 2022-05-18 18:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  2022-05-18 19:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-18 18:26 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, May 17, 2022 at 5:22 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, May 17, 2022 at 04:39:18PM -0400, Josef Bacik wrote:
> > What I *think* happened is you essentially lost random blocks from 1
> > transaction.  This isn't particularly harmful, except you had a chunk
> > allocation that happened, so you lost some of the chunk root.  I
> > didn't realize this is what was happening until now.  If I had I
> > probably could have pieced together these dev extents into chunks, and
> > then all those blocks that we had that didn't map to a chunk probably
> > would have been fine, because they were in the range that we lost.
> >
> > It's not that you lost a lot of blocks, you just lost a few really,
> > really important blocks, and then I failed to recognize what happened
> > so I threw out more than I should have, plus all the other little
> > wonkiness from experimental recovery tools.  Thanks,
>
> Right, which is totally understandable. I'm still glad I have backups
> (waiting to restore), but I very much appreciate your work right now as
> it's fixing the bigger problem of recovering an unmountable filesystem.
>
> Ultimately I don't care as much about how much data is lost, but I very
> much care about btrfs check repair getting the filesystem back to a
> mountable state, so I've been more than happy helping you get to that
> state and I'm very thankful for all the time you spent on this so far.
> Hopefully we're almost there :)
>
> For extra points, files that have corrupted checksums, I'd love for them
> to be renamed to file.corrupted, so that someone who cares about partial
> data, can get some data back, and that you can recover much faster with
> rsync if it only has to look for missing files, as opposed to running a
> checksum on all the data on both sides, which would take ages.
> I know in this case, I may end up with some amount of files silently
> corrupted since we rebuilt the checksum table, not sure if there is a
> way to do this like I described or some similar way, in the future.
>
> I see you pushed another change, trying it now.
>
> I have the full output saved if need be.
>
> Device extent[1, 14812306210816, 1073741824] didn't find the relative chunk.
> Device extent[1, 9544528822272, 1073741824] didn't find the relative chunk.
> Device extent[1, 11231377227776, 1073741824] didn't find the relative chunk.
> Device extent[1, 11406397145088, 1073741824] didn't find the relative chunk.
> Device extent[1, 11416060821504, 1073741824] didn't find the relative chunk.
> Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
> Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
> incorrect local backref count on 2952871936 parent 13576823652352 owner 0 offset 0 found 0 wanted 1 back 0x55db601ea230
> backref disk bytenr does not match extent record, bytenr=2952871936, ref bytenr=0
> data backref 2952871936 root 11223 owner 258 offset 0 num_refs 0 not found in extent tree
> incorrect local backref count on 2952871936 root 11223 owner 258 offset 0 found 1 wanted 0 back 0x55db80f5c6a0
> backpointer mismatch on [2952871936 262144]
> repair deleting extent record: key [2952871936,168,262144]
> adding new data backref on 2952871936 root 1 owner 258 offset 0 found 1
> adding new data backref on 2952871936 root 11223 owner 258 offset 0 found 1
> Repaired extent references for 2952871936
> incorrect local backref count on 4156227584 parent 13576823652352 owner 0 offset 0 found 0 wanted 1 back 0x55db607ad1f0
> backref disk bytenr does not match extent record, bytenr=4156227584, ref bytenr=0
> data backref 4156227584 root 11223 owner 259 offset 0 num_refs 0 not found in extent tree
> incorrect local backref count on 4156227584 root 11223 owner 259 offset 0 found 1 wanted 0 back 0x55db80f5c570
> backpointer mismatch on [4156227584 262144]
> repair deleting extent record: key [4156227584,168,262144]
> adding new data backref on 4156227584 root 1 owner 259 offset 0 found 1
> adding new data backref on 4156227584 root 11223 owner 259 offset 0 found 1
> Repaired extent references for 4156227584
> (...)
> adding new data backref on 10449820549120 root 1 owner 256 offset 0 found 1
> adding new data backref on 10449820549120 root 11223 owner 256 offset 0 found 1
> Repaired extent references for 10449820549120
> super bytes used 14180042240000 mismatches actual used 14180042256384
> Invalid key type(ROOT_ITEM) found in root(11223)
> ignoring invalid key
> Invalid key type(ROOT_ITEM) found in root(11223)
> ignoring invalid key
> Invalid key type(ROOT_ITEM) found in root(11223)
> ignoring invalid key
> Invalid key type(ROOT_ITEM) found in root(11223)
> ignoring invalid key
> Invalid key type(ROOT_ITEM) found in root(11223)
> ignoring invalid key
> Block group[4523166793728, 1073741824] (flags = 1) didn't find the relative chunk.
> Block group[4524240535552, 1073741824] (flags = 1) didn't find the relative chunk.
> Block group[4525314277376, 1073741824] (flags = 1) didn't find the relative chunk.
> Block group[4526388019200, 1073741824] (flags = 1) didn't find the relative chunk.
> Block group[4527461761024, 1073741824] (flags = 1) didn't find the relative chunk.

Hrm crud, I've fixed this, but you may have to re-run the init's.  Start with

btrfs check --repair <device>

and then see if it works.  If not do

btrfs rescue init-extent-tree <device>
btrfs rescue init-csum-tree <device>
btrfs check <device>

and then you should be good to go.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-18 18:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
@ 2022-05-18 19:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  2022-05-18 19:17                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-18 19:12 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, May 18, 2022 at 02:26:36PM -0400, Josef Bacik wrote:
> Hrm crud, I've fixed this, but you may have to re-run the init's.  Start with
> 
> btrfs check --repair <device>
> 
> and then see if it works.  If not do
 
Block group[15360476577792, 1073741824] (flags = 1) didn't find the relative chunk.
Block group[15361550319616, 1073741824] (flags = 1) didn't find the relative chunk.
Block group[15362624061440, 1073741824] (flags = 1) didn't find the relative chunk.
Block group[15363697803264, 1073741824] (flags = 1) didn't find the relative chunk.
Block group[15364771545088, 1073741824] (flags = 1) didn't find the relative chunk.
Block group[15365845286912, 1073741824] (flags = 1) didn't find the relative chunk.
Block group[15366919028736, 1073741824] (flags = 1) didn't find the relative chunk.
failed to repair damaged filesystem, aborting

Starting repair.
Opening filesystem to check...
JOSEF: root 9
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e

> btrfs rescue init-extent-tree <device>

Whatever we did may have caused a bunch of new files to be invalid and have to be deleted.

searching 159785 for bad extents
processed 11304960 of 108429312 possible bytes, 10%
Found an extent we don't have a block group for in the file
file
Deleting [11142, 108, 1020231680] root 6781246029824 path top 6781246029824 top slot 2 leaf 3861741830144 slot 109

searching 159785 for bad extents
processed 11304960 of 108429312 possible bytes, 10%
Found an extent we don't have a block group for in the file
file
Deleting [11142, 108, 1020280832] root 6781246046208 path top 6781246046208 top slot 2 leaf 3861741846528 slot 109

searching 159785 for bad extents
processed 11304960 of 108429312 possible bytes, 10%
Found an extent we don't have a block group for in the file
file
Deleting [11142, 108, 1154498560] root 6781246029824 path top 6781246029824 top slot 2 leaf 3861741830144 slot 109

searching 159785 for bad extents
processed 11304960 of 108429312 possible bytes, 10%
Found an extent we don't have a block group for in the file
file
Deleting [11142, 108, 1288716288] root 6781246046208 path top 6781246046208 top slot 2 leaf 3861741846528 slot 10

Mmmh, it's deleted 2500 files already, I just stopped it
gargamel:~# grep -c Deleting /mnt/btrfs_space/ri1
2507

Safe to continue?

> btrfs rescue init-csum-tree <device>
> btrfs check <device>

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-18 19:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
@ 2022-05-18 19:17                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  2022-05-19 22:28                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-18 19:17 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, May 18, 2022 at 3:12 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, May 18, 2022 at 02:26:36PM -0400, Josef Bacik wrote:
> > Hrm crud, I've fixed this, but you may have to re-run the init's.  Start with
> >
> > btrfs check --repair <device>
> >
> > and then see if it works.  If not do
>
> Block group[15360476577792, 1073741824] (flags = 1) didn't find the relative chunk.
> Block group[15361550319616, 1073741824] (flags = 1) didn't find the relative chunk.
> Block group[15362624061440, 1073741824] (flags = 1) didn't find the relative chunk.
> Block group[15363697803264, 1073741824] (flags = 1) didn't find the relative chunk.
> Block group[15364771545088, 1073741824] (flags = 1) didn't find the relative chunk.
> Block group[15365845286912, 1073741824] (flags = 1) didn't find the relative chunk.
> Block group[15366919028736, 1073741824] (flags = 1) didn't find the relative chunk.
> failed to repair damaged filesystem, aborting
>
> Starting repair.
> Opening filesystem to check...
> JOSEF: root 9
> Checking filesystem on /dev/mapper/dshelf1
> UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
>
> > btrfs rescue init-extent-tree <device>
>
> Whatever we did may have caused a bunch of new files to be invalid and have to be deleted.
>
> searching 159785 for bad extents
> processed 11304960 of 108429312 possible bytes, 10%
> Found an extent we don't have a block group for in the file
> file
> Deleting [11142, 108, 1020231680] root 6781246029824 path top 6781246029824 top slot 2 leaf 3861741830144 slot 109
>
> searching 159785 for bad extents
> processed 11304960 of 108429312 possible bytes, 10%
> Found an extent we don't have a block group for in the file
> file
> Deleting [11142, 108, 1020280832] root 6781246046208 path top 6781246046208 top slot 2 leaf 3861741846528 slot 109
>
> searching 159785 for bad extents
> processed 11304960 of 108429312 possible bytes, 10%
> Found an extent we don't have a block group for in the file
> file
> Deleting [11142, 108, 1154498560] root 6781246029824 path top 6781246029824 top slot 2 leaf 3861741830144 slot 109
>
> searching 159785 for bad extents
> processed 11304960 of 108429312 possible bytes, 10%
> Found an extent we don't have a block group for in the file
> file
> Deleting [11142, 108, 1288716288] root 6781246046208 path top 6781246046208 top slot 2 leaf 3861741846528 slot 10
>
> Mmmh, it's deleted 2500 files already, I just stopped it
> gargamel:~# grep -c Deleting /mnt/btrfs_space/ri1
> 2507
>
> Safe to continue?

Yes sorry I meant to say that.  Because we have these dangling block
groups we'll suddenly have a bunch of files that no longer are
mappable and we'll need to delete them.  Looks to be about 7gib of
block groups so you're going to lose that stuff, it's going to be a
while but it's expected.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-18 19:17                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
@ 2022-05-19 22:28                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  2022-05-24  1:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-19 22:28 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, May 18, 2022 at 03:17:55PM -0400, Josef Bacik wrote:
> Yes sorry I meant to say that.  Because we have these dangling block
> groups we'll suddenly have a bunch of files that no longer are
> mappable and we'll need to delete them.  Looks to be about 7gib of
> block groups so you're going to lose that stuff, it's going to be a
> while but it's expected.  Thanks,
> 
So, it's definitely deleting a lot

I think I'm at 81%

searching 159785 for bad extents
processed 78446592 of 95879168 possible bytes, 81%
Found an extent we don't have a block group for in the file
Performances/Magic/Diversion 08022019.mkv
Deleting [70879, 108, 10708312064] root 6781246029824 path top 6781246029824 top slot 19 leaf 10678930079744 slot 34

gargamel:~# grep -c Deleting /mnt/btrfs_space/ri1
 149583

Ok, that's a lot of files, but let's see if it finishes

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-19 22:28                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
@ 2022-05-24  1:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-05-24 18:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-24  1:13 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, May 19, 2022 at 03:28:55PM -0700, Marc MERLIN wrote:
> On Wed, May 18, 2022 at 03:17:55PM -0400, Josef Bacik wrote:
> > Yes sorry I meant to say that.  Because we have these dangling block
> > groups we'll suddenly have a bunch of files that no longer are
> > mappable and we'll need to delete them.  Looks to be about 7gib of
> > block groups so you're going to lose that stuff, it's going to be a
> > while but it's expected.  Thanks,
> > 
> So, it's definitely deleting a lot
> 
> I think I'm at 81%
> 
> searching 159785 for bad extents
> processed 78446592 of 95879168 possible bytes, 81%
> Found an extent we don't have a block group for in the file
> Performances/Magic/Diversion 08022019.mkv
> Deleting [70879, 108, 10708312064] root 6781246029824 path top 6781246029824 top slot 19 leaf 10678930079744 slot 34
> 
> gargamel:~# grep -c Deleting /mnt/btrfs_space/ri1
>  149583
> 
> Ok, that's a lot of files, but let's see if it finishes

mmmh, so I'm not sure if it's going to finish soon? still going after 5
days.
searching 162632 for bad extents
processed 78446592 of 99090432 possible bytes, 79%
Found an extent we don't have a block group for in the file
file
Deleting [70879, 108, 9305350144] root 6781246177280 path top 6781246177280 top slot 19 leaf 782788263936 slot 0

searching 162632 for bad extents
processed 78446592 of 99090432 possible bytes, 79%
Found an extent we don't have a block group for in the file
file
Deleting [70879, 108, 9305358336] root 6781246128128 path top 6781246128128 top slot 19 leaf 782788296704 slot 0

searching 162632 for bad extents
processed 78446592 of 99090432 possible bytes, 79%
Found an extent we don't have a block group for in the file
file
Deleting [70879, 108, 9305411584] root 6781246177280 path top 6781246177280 top slot 19 leaf 782788263936 slot 0


gargamel:~# grep -c Deleting /mnt/btrfs_space/ri1
620577

over 600K files deleted, it's still moving forward though.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-24  1:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-05-24 18:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-05-24 19:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-24 18:26 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 23, 2022 at 9:13 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, May 19, 2022 at 03:28:55PM -0700, Marc MERLIN wrote:
> > On Wed, May 18, 2022 at 03:17:55PM -0400, Josef Bacik wrote:
> > > Yes sorry I meant to say that.  Because we have these dangling block
> > > groups we'll suddenly have a bunch of files that no longer are
> > > mappable and we'll need to delete them.  Looks to be about 7gib of
> > > block groups so you're going to lose that stuff, it's going to be a
> > > while but it's expected.  Thanks,
> > >
> > So, it's definitely deleting a lot
> >
> > I think I'm at 81%
> >
> > searching 159785 for bad extents
> > processed 78446592 of 95879168 possible bytes, 81%
> > Found an extent we don't have a block group for in the file
> > Performances/Magic/Diversion 08022019.mkv
> > Deleting [70879, 108, 10708312064] root 6781246029824 path top 6781246029824 top slot 19 leaf 10678930079744 slot 34
> >
> > gargamel:~# grep -c Deleting /mnt/btrfs_space/ri1
> >  149583
> >
> > Ok, that's a lot of files, but let's see if it finishes
>
> mmmh, so I'm not sure if it's going to finish soon? still going after 5
> days.
> searching 162632 for bad extents
> processed 78446592 of 99090432 possible bytes, 79%
> Found an extent we don't have a block group for in the file
> file
> Deleting [70879, 108, 9305350144] root 6781246177280 path top 6781246177280 top slot 19 leaf 782788263936 slot 0
>
> searching 162632 for bad extents
> processed 78446592 of 99090432 possible bytes, 79%
> Found an extent we don't have a block group for in the file
> file
> Deleting [70879, 108, 9305358336] root 6781246128128 path top 6781246128128 top slot 19 leaf 782788296704 slot 0
>
> searching 162632 for bad extents
> processed 78446592 of 99090432 possible bytes, 79%
> Found an extent we don't have a block group for in the file
> file
> Deleting [70879, 108, 9305411584] root 6781246177280 path top 6781246177280 top slot 19 leaf 782788263936 slot 0
>
>
> gargamel:~# grep -c Deleting /mnt/btrfs_space/ri1
> 620577
>
> over 600K files deleted, it's still moving forward though.

So it's 600k extents, not files.  Here you have the same file, just
different offsets

Deleting [70879, 108, 9305350144] root 6781246177280 path top
6781246177280 top slot 19 leaf 782788263936 slot 0
Deleting [70879, 108, 9305411584] root 6781246177280 path top
6781246177280 top slot 19 leaf 782788263936 slot 0

I'm really wishing I had noticed this earlier, it would have been
faster to put together the old block groups than to wait for this,
sorry about that.

Right now I feel like absolute crap, my kids gave me COVID so I've
been fluctuating between dying and feeling ok.  Tomorrow if I'm
feeling better I'll take a look at the chunk restore code and see if I
can't just put the tree back the way it was easily.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-24 18:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-05-24 19:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-05-25 14:35                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-24 19:13 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, May 24, 2022 at 02:26:11PM -0400, Josef Bacik wrote:
> So it's 600k extents, not files.  Here you have the same file, just
> different offsets
 
True.

> Right now I feel like absolute crap, my kids gave me COVID so I've
> been fluctuating between dying and feeling ok.  Tomorrow if I'm

Sorry :( hope you get better soon

> feeling better I'll take a look at the chunk restore code and see if I
> can't just put the tree back the way it was easily.  Thanks,

As long as it's not deleting stuff it shouldn't delete, I'll keep it
running for now, or should I stop it?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-24 19:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-05-25 14:35                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-05-26 17:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-25 14:35 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, May 24, 2022 at 3:13 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, May 24, 2022 at 02:26:11PM -0400, Josef Bacik wrote:
> > So it's 600k extents, not files.  Here you have the same file, just
> > different offsets
>
> True.
>
> > Right now I feel like absolute crap, my kids gave me COVID so I've
> > been fluctuating between dying and feeling ok.  Tomorrow if I'm
>
> Sorry :( hope you get better soon
>
> > feeling better I'll take a look at the chunk restore code and see if I
> > can't just put the tree back the way it was easily.  Thanks,
>
> As long as it's not deleting stuff it shouldn't delete, I'll keep it
> running for now, or should I stop it?
>

At this point let's stop it and try and get the missing chunks back.
Looking at the chunk rescue code it looks like it should do the
correct thing, can you do

btrfs rescue chunk-recover <device>

and let me know what that does?  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-25 14:35                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-05-26 17:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-05-26 17:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-26 17:10 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, May 25, 2022 at 10:35:22AM -0400, Josef Bacik wrote:
> At this point let's stop it and try and get the missing chunks back.
> Looking at the chunk rescue code it looks like it should do the
> correct thing, can you do

I stopped it here, can't tell how far it was after almost a week of running:
 
searching 163316 for bad extents
processed 74514432 of 108576768 possible bytes, 68%
Found an extent we don't have a block group for in the file
History/JeanMerlin_Picts/VACANCES/0320-052633.jpg
Deleting [58726, 108, 0] root 6781246275584 path top 6781246275584 top slot 18 leaf 11160503386112 slot 80

searching 163316 for bad extents
processed 74514432 of 108576768 possible bytes, 68%
Found an extent we don't have a block group for in the file
History/JeanMerlin_Picts/VACANCES/0320-052720.jpg
Deleting [58727, 108, 0] root 6781246226432 path top 6781246226432 top slot 18 leaf 11160503353344 slot 82

searching 163316 for bad extents
processed 74514432 of 108576768 possible bytes, 68%
Found an extent we don't have a block group for in the file
History/JeanMerlin_Picts/VACANCES/0320-053405.jpg
Deleting [58728, 108, 0] root 6781246275584 path top 6781246275584 top slot 18 leaf 11160503386112 slot 84

searching 163316 for bad extents
processed ^Z515008 of 108576768 possible bytes, 67%

> btrfs rescue chunk-recover <device>

Took close to a day to run, and now I have this:

./btrfs rescue chunk-recover /dev/mapper/dshelf1
Scanning: DONE in dev0                          
JOSEF: root 9
Couldn't find the last root for 8
We are going to rebuild the chunk tree on disk, it might destroy the old metadata on the disk, Are you sure? [y/N]: 

Do I say 'y' ?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-26 17:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-05-26 17:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-05-26 17:31                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-26 17:12 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, May 26, 2022 at 1:10 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, May 25, 2022 at 10:35:22AM -0400, Josef Bacik wrote:
> > At this point let's stop it and try and get the missing chunks back.
> > Looking at the chunk rescue code it looks like it should do the
> > correct thing, can you do
>
> I stopped it here, can't tell how far it was after almost a week of running:
>
> searching 163316 for bad extents
> processed 74514432 of 108576768 possible bytes, 68%
> Found an extent we don't have a block group for in the file
> History/JeanMerlin_Picts/VACANCES/0320-052633.jpg
> Deleting [58726, 108, 0] root 6781246275584 path top 6781246275584 top slot 18 leaf 11160503386112 slot 80
>
> searching 163316 for bad extents
> processed 74514432 of 108576768 possible bytes, 68%
> Found an extent we don't have a block group for in the file
> History/JeanMerlin_Picts/VACANCES/0320-052720.jpg
> Deleting [58727, 108, 0] root 6781246226432 path top 6781246226432 top slot 18 leaf 11160503353344 slot 82
>
> searching 163316 for bad extents
> processed 74514432 of 108576768 possible bytes, 68%
> Found an extent we don't have a block group for in the file
> History/JeanMerlin_Picts/VACANCES/0320-053405.jpg
> Deleting [58728, 108, 0] root 6781246275584 path top 6781246275584 top slot 18 leaf 11160503386112 slot 84
>
> searching 163316 for bad extents
> processed ^Z515008 of 108576768 possible bytes, 67%
>
> > btrfs rescue chunk-recover <device>
>
> Took close to a day to run, and now I have this:
>
> ./btrfs rescue chunk-recover /dev/mapper/dshelf1
> Scanning: DONE in dev0
> JOSEF: root 9
> Couldn't find the last root for 8
> We are going to rebuild the chunk tree on disk, it might destroy the old metadata on the disk, Are you sure? [y/N]:
>
> Do I say 'y' ?
>

Yup it should have found all the best chunks, lets let it do its thing.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-26 17:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-05-26 17:31                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-05-26 17:44                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-26 17:31 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, May 26, 2022 at 01:12:03PM -0400, Josef Bacik wrote:
> > ./btrfs rescue chunk-recover /dev/mapper/dshelf1
> > Scanning: DONE in dev0
> > JOSEF: root 9
> > Couldn't find the last root for 8
> > We are going to rebuild the chunk tree on disk, it might destroy the old metadata on the disk, Are you sure? [y/N]:
> >
> > Do I say 'y' ?
> 
> Yup it should have found all the best chunks, lets let it do its thing.  Thanks,
 
Done.

what next?


gargamel:/var/local/src/btrfs-progs-josefbacik# fg
./btrfs rescue chunk-recover /dev/mapper/dshelf1
Scanning: DONE in dev0                          
JOSEF: root 9
Couldn't find the last root for 8
We are going to rebuild the chunk tree on disk, it might destroy the old metadata on the disk, Are you sure? [y/N]: y
Chunk tree recovered successfully
gargamel:/var/local/src/btrfs-progs-josefbacik# 

I still have this that ran almost a week:
gargamel:/var/local/src/btrfs-progs-josefbacik# jobs
[1]+  Stopped                 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 2>&1 | tee /mnt/btrfs_space/ri1

I assume it's invalidated and I should kill it?
If so, do I start the same thing again?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-26 17:31                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-05-26 17:44                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-05-26 18:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-26 17:44 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, May 26, 2022 at 1:31 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, May 26, 2022 at 01:12:03PM -0400, Josef Bacik wrote:
> > > ./btrfs rescue chunk-recover /dev/mapper/dshelf1
> > > Scanning: DONE in dev0
> > > JOSEF: root 9
> > > Couldn't find the last root for 8
> > > We are going to rebuild the chunk tree on disk, it might destroy the old metadata on the disk, Are you sure? [y/N]:
> > >
> > > Do I say 'y' ?
> >
> > Yup it should have found all the best chunks, lets let it do its thing.  Thanks,
>
> Done.
>
> what next?
>
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# fg
> ./btrfs rescue chunk-recover /dev/mapper/dshelf1
> Scanning: DONE in dev0
> JOSEF: root 9
> Couldn't find the last root for 8
> We are going to rebuild the chunk tree on disk, it might destroy the old metadata on the disk, Are you sure? [y/N]: y
> Chunk tree recovered successfully
> gargamel:/var/local/src/btrfs-progs-josefbacik#
>
> I still have this that ran almost a week:
> gargamel:/var/local/src/btrfs-progs-josefbacik# jobs
> [1]+  Stopped                 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 2>&1 | tee /mnt/btrfs_space/ri1
>
> I assume it's invalidated and I should kill it?
> If so, do I start the same thing again?
>

Kill it and start it again, hopefully this time we have all the chunks
and the init should be quick like it was before.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-26 17:44                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-05-26 18:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-05-26 18:54                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-26 18:12 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, May 26, 2022 at 01:44:00PM -0400, Josef Bacik wrote:
> > I still have this that ran almost a week:
> > gargamel:/var/local/src/btrfs-progs-josefbacik# jobs
> > [1]+  Stopped                 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 2>&1 | tee /mnt/btrfs_space/ri1
> >
> > I assume it's invalidated and I should kill it?
> > If so, do I start the same thing again?
> >
> 
> Kill it and start it again, hopefully this time we have all the chunks
> and the init should be quick like it was before.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 2>&1 | tee /mnt/btrfs_space/ri1
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
Csum didn't match
ERROR: cannot read chunk root
WTF???
ERROR: open ctree failed, try btrfs rescue tree-recover
Init extent tree failed

Do I do
./btrfs rescue tree-recover --init-extent-tree /dev/mapper/dshelf1
or
./btrfs rescue tree-recover /dev/mapper/dshelf1

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-26 18:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-05-26 18:54                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-05-26 19:15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-26 18:54 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, May 26, 2022 at 2:12 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, May 26, 2022 at 01:44:00PM -0400, Josef Bacik wrote:
> > > I still have this that ran almost a week:
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# jobs
> > > [1]+  Stopped                 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 2>&1 | tee /mnt/btrfs_space/ri1
> > >
> > > I assume it's invalidated and I should kill it?
> > > If so, do I start the same thing again?
> > >
> >
> > Kill it and start it again, hopefully this time we have all the chunks
> > and the init should be quick like it was before.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 2>&1 | tee /mnt/btrfs_space/ri1
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> Csum didn't match
> ERROR: cannot read chunk root
> WTF???
> ERROR: open ctree failed, try btrfs rescue tree-recover
> Init extent tree failed
>
> Do I do
> ./btrfs rescue tree-recover --init-extent-tree /dev/mapper/dshelf1
> or
> ./btrfs rescue tree-recover /dev/mapper/dshelf1

Tree-recover first, lord I'm tired of our tools randomly not updating
root pointers.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-26 18:54                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-05-26 19:15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-05-26 19:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-26 19:15 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, May 26, 2022 at 02:54:47PM -0400, Josef Bacik wrote:
> > Do I do
> > ./btrfs rescue tree-recover --init-extent-tree /dev/mapper/dshelf1
> > or
> > ./btrfs rescue tree-recover /dev/mapper/dshelf1
> 
> Tree-recover first, lord I'm tired of our tools randomly not updating
> root pointers.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 
ERROR: cannot read chunk root
WTF???
ERROR: open ctree failed
Tree recover failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-26 19:15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-05-26 19:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-05-26 21:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-26 19:55 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, May 26, 2022 at 3:15 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, May 26, 2022 at 02:54:47PM -0400, Josef Bacik wrote:
> > > Do I do
> > > ./btrfs rescue tree-recover --init-extent-tree /dev/mapper/dshelf1
> > > or
> > > ./btrfs rescue tree-recover /dev/mapper/dshelf1
> >
> > Tree-recover first, lord I'm tired of our tools randomly not updating
> > root pointers.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> ERROR: cannot read chunk root
> WTF???
> ERROR: open ctree failed
> Tree recover failed

Sigh, I've pushed new code, build and run

./btrfs-find-root -o 3 <device>

and let me know what it says.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-26 19:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-05-26 21:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-05-26 23:23                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-05-27  1:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 2 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-05-26 21:39 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, May 26, 2022 at 03:55:29PM -0400, Josef Bacik wrote:
> On Thu, May 26, 2022 at 3:15 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Thu, May 26, 2022 at 02:54:47PM -0400, Josef Bacik wrote:
> > > > Do I do
> > > > ./btrfs rescue tree-recover --init-extent-tree /dev/mapper/dshelf1
> > > > or
> > > > ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > >
> > > Tree-recover first, lord I'm tired of our tools randomly not updating
> > > root pointers.  Thanks,
> >
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > ERROR: cannot read chunk root
> > WTF???
> > ERROR: open ctree failed
> > Tree recover failed
> 
> Sigh, I've pushed new code, build and run
> 
> ./btrfs-find-root -o 3 <device>
> 
> and let me know what it says.  Thanks,

I thought we were getting so close, but it seems we'e made a few steps
back :-/

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 3 /dev/mapper/dshelf1
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
Csum didn't match
ERROR: cannot read chunk root
WTF???
ERROR: open ctree failed

At some point, if the FS is starting to look like it was trashed to
start with, or kind of trashed now after some of the recovery attempts,
let me know and I'll wipe and restore.
That said, if there is still data useful to improving your tools, I'm
game for a bit more, although if we hit the 2 months mark since it went
down, I'll have to eventuallly recover :)

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-26 21:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-05-26 23:23                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-05-27  1:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  1 sibling, 0 replies; 479+ messages in thread
From: Josef Bacik @ 2022-05-26 23:23 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, May 26, 2022 at 5:39 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, May 26, 2022 at 03:55:29PM -0400, Josef Bacik wrote:
> > On Thu, May 26, 2022 at 3:15 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Thu, May 26, 2022 at 02:54:47PM -0400, Josef Bacik wrote:
> > > > > Do I do
> > > > > ./btrfs rescue tree-recover --init-extent-tree /dev/mapper/dshelf1
> > > > > or
> > > > > ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > > >
> > > > Tree-recover first, lord I'm tired of our tools randomly not updating
> > > > root pointers.  Thanks,
> > >
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > > ERROR: cannot read chunk root
> > > WTF???
> > > ERROR: open ctree failed
> > > Tree recover failed
> >
> > Sigh, I've pushed new code, build and run
> >
> > ./btrfs-find-root -o 3 <device>
> >
> > and let me know what it says.  Thanks,
>
> I thought we were getting so close, but it seems we'e made a few steps
> back :-/
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 3 /dev/mapper/dshelf1
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> Csum didn't match
> ERROR: cannot read chunk root
> WTF???
> ERROR: open ctree failed
>
> At some point, if the FS is starting to look like it was trashed to
> start with, or kind of trashed now after some of the recovery attempts,
> let me know and I'll wipe and restore.
> That said, if there is still data useful to improving your tools, I'm
> game for a bit more, although if we hit the 2 months mark since it went
> down, I'll have to eventuallly recover :)
>

This is approaching insanity a little bit, but we're relatively close.
I fixed the flags, hopefully it'll work this time.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-26 21:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-05-26 23:23                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-05-27  1:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  2022-05-27 18:35                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-27  1:16 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, May 26, 2022 at 02:39:24PM -0700, Marc MERLIN wrote:
> I thought we were getting so close, but it seems we'e made a few steps
> back :-/
> 
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 3 /dev/mapper/dshelf1
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> Csum didn't match
> ERROR: cannot read chunk root
> WTF???
> ERROR: open ctree failed
> 
> At some point, if the FS is starting to look like it was trashed to
> start with, or kind of trashed now after some of the recovery attempts,
> let me know and I'll wipe and restore.
> That said, if there is still data useful to improving your tools, I'm
> game for a bit more, although if we hit the 2 months mark since it went
> down, I'll have to eventuallly recover :)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 3 /dev/mapper/dshelf1
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
Csum didn't match
WARNING: cannot read chunk root, continue anyway
Superblock thinks the generation is 1739781
Superblock thinks the level is 1
Well block 22593536(gen: 1590219 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
Well block 22560768(gen: 1590217 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
Well block 22577152(gen: 1586277 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
Well block 22495232(gen: 1572124 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
Well block 22528000(gen: 1572115 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
Well block 22446080(gen: 1571791 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
Well block 22478848(gen: 1561557 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
Well block 22544384(gen: 1556078 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
Well block 22511616(gen: 1555799 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
Well block 22609920(gen: 1551635 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
Well block 22462464(gen: 1479229 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
gargamel:/var/local/src/btrfs-progs-josefbacik# 

Is that good?
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-27  1:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
@ 2022-05-27 18:35                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  2022-05-27 23:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-27 18:35 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, May 26, 2022 at 9:16 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, May 26, 2022 at 02:39:24PM -0700, Marc MERLIN wrote:
> > I thought we were getting so close, but it seems we'e made a few steps
> > back :-/
> >
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 3 /dev/mapper/dshelf1
> > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > Csum didn't match
> > ERROR: cannot read chunk root
> > WTF???
> > ERROR: open ctree failed
> >
> > At some point, if the FS is starting to look like it was trashed to
> > start with, or kind of trashed now after some of the recovery attempts,
> > let me know and I'll wipe and restore.
> > That said, if there is still data useful to improving your tools, I'm
> > game for a bit more, although if we hit the 2 months mark since it went
> > down, I'll have to eventuallly recover :)
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 3 /dev/mapper/dshelf1
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> Csum didn't match
> WARNING: cannot read chunk root, continue anyway
> Superblock thinks the generation is 1739781
> Superblock thinks the level is 1
> Well block 22593536(gen: 1590219 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
> Well block 22560768(gen: 1590217 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
> Well block 22577152(gen: 1586277 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
> Well block 22495232(gen: 1572124 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
> Well block 22528000(gen: 1572115 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
> Well block 22446080(gen: 1571791 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
> Well block 22478848(gen: 1561557 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
> Well block 22544384(gen: 1556078 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
> Well block 22511616(gen: 1555799 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
> Well block 22609920(gen: 1551635 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
> Well block 22462464(gen: 1479229 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1
> gargamel:/var/local/src/btrfs-progs-josefbacik#
>
> Is that good?

I'm augmenting my tree-recover tool to go and find any missing chunks
and add them in, which is what the chunk recover thing was supposed to
do.  This is going to take a bit, but should be the last piece.
Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-27 18:35                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
@ 2022-05-27 23:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  2022-05-28  0:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-27 23:26 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Fri, May 27, 2022 at 02:35:05PM -0400, Josef Bacik wrote:
> I'm augmenting my tree-recover tool to go and find any missing chunks
> and add them in, which is what the chunk recover thing was supposed to
> do.  This is going to take a bit, but should be the last piece.

And by that you mean you're working on it and will tell me when it's
ready to pull, or did you forget git push?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-27 23:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
@ 2022-05-28  0:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  2022-05-28 20:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-28  0:13 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Fri, May 27, 2022 at 7:26 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Fri, May 27, 2022 at 02:35:05PM -0400, Josef Bacik wrote:
> > I'm augmenting my tree-recover tool to go and find any missing chunks
> > and add them in, which is what the chunk recover thing was supposed to
> > do.  This is going to take a bit, but should be the last piece.
>
> And by that you mean you're working on it and will tell me when it's
> ready to pull, or did you forget git push?
>

Still typing, just didn't want you to think I'd disappeared.  I'll let
you know when you can pull.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-28  0:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
@ 2022-05-28 20:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-05-28 22:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-28 20:08 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Fri, May 27, 2022 at 8:13 PM Josef Bacik <josef@toxicpanda.com> wrote:
>
> On Fri, May 27, 2022 at 7:26 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Fri, May 27, 2022 at 02:35:05PM -0400, Josef Bacik wrote:
> > > I'm augmenting my tree-recover tool to go and find any missing chunks
> > > and add them in, which is what the chunk recover thing was supposed to
> > > do.  This is going to take a bit, but should be the last piece.
> >
> > And by that you mean you're working on it and will tell me when it's
> > ready to pull, or did you forget git push?
> >
>
> Still typing, just didn't want you to think I'd disappeared.  I'll let
> you know when you can pull.  Thanks,
>

Sorry my ability to think isn't doing so great right now.  I've wired
up the detection stuff, but it won't actually fix anything yet.  I
want to make sure I've got detection part right before I go messing
with the file system.  If you can pull and build and then run

btrfs rescue recover-chunks <device>

and capture the output that would be great.  Hopefully this shows the
missing block groups and I can just copy them into place.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-28 20:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-05-28 22:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-05-29  1:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-28 22:56 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sat, May 28, 2022 at 04:08:25PM -0400, Josef Bacik wrote:
> Sorry my ability to think isn't doing so great right now.  I've wired
> up the detection stuff, but it won't actually fix anything yet.  I
> want to make sure I've got detection part right before I go messing
> with the file system.  If you can pull and build and then run
> 
> btrfs rescue recover-chunks <device>
> 
> and capture the output that would be great.  Hopefully this shows the
> missing block groups and I can just copy them into place.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
Csum didn't match
ERROR: cannot read chunk root
WTF???
ERROR: open ctree failed, try btrfs rescue tree-recover
Recover chunks tree failed
gargamel:/var/local/src/btrfs-progs-josefbacik# 

So, should I do what it says?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-28 22:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-05-29  1:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-05-29  3:51                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-29  1:00 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sat, May 28, 2022 at 6:56 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sat, May 28, 2022 at 04:08:25PM -0400, Josef Bacik wrote:
> > Sorry my ability to think isn't doing so great right now.  I've wired
> > up the detection stuff, but it won't actually fix anything yet.  I
> > want to make sure I've got detection part right before I go messing
> > with the file system.  If you can pull and build and then run
> >
> > btrfs rescue recover-chunks <device>
> >
> > and capture the output that would be great.  Hopefully this shows the
> > missing block groups and I can just copy them into place.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> Csum didn't match
> ERROR: cannot read chunk root
> WTF???
> ERROR: open ctree failed, try btrfs rescue tree-recover
> Recover chunks tree failed
> gargamel:/var/local/src/btrfs-progs-josefbacik#
>
> So, should I do what it says?

Oh yeah duh sorry, I was adding it to tree-recover but decided it was
a bad place for it.  Go ahead and run tree-recover and then do the
recover-chunks.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-29  1:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-05-29  3:51                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-05-29 15:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-29  3:51 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sat, May 28, 2022 at 09:00:00PM -0400, Josef Bacik wrote:
> On Sat, May 28, 2022 at 6:56 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sat, May 28, 2022 at 04:08:25PM -0400, Josef Bacik wrote:
> > > Sorry my ability to think isn't doing so great right now.  I've wired
> > > up the detection stuff, but it won't actually fix anything yet.  I
> > > want to make sure I've got detection part right before I go messing
> > > with the file system.  If you can pull and build and then run
> > >
> > > btrfs rescue recover-chunks <device>
> > >
> > > and capture the output that would be great.  Hopefully this shows the
> > > missing block groups and I can just copy them into place.  Thanks,
> >
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > Csum didn't match
> > ERROR: cannot read chunk root
> > WTF???
> > ERROR: open ctree failed, try btrfs rescue tree-recover
> > Recover chunks tree failed
> > gargamel:/var/local/src/btrfs-progs-josefbacik#
> >
> > So, should I do what it says?
> 
> Oh yeah duh sorry, I was adding it to tree-recover but decided it was
> a bad place for it.  Go ahead and run tree-recover and then do the
> recover-chunks.  Thanks,

:(

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
ERROR: cannot read chunk root
WTF???
ERROR: open ctree failed
Tree recover failed

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
Csum didn't match
ERROR: cannot read chunk root
WTF???
ERROR: open ctree failed, try btrfs rescue tree-recover
Recover chunks tree failed
gargamel:/var/local/src/btrfs-progs-josefbacik# 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-29  3:51                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-05-29 15:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-05-29 15:33                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-29 15:00 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sat, May 28, 2022 at 11:51 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sat, May 28, 2022 at 09:00:00PM -0400, Josef Bacik wrote:
> > On Sat, May 28, 2022 at 6:56 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sat, May 28, 2022 at 04:08:25PM -0400, Josef Bacik wrote:
> > > > Sorry my ability to think isn't doing so great right now.  I've wired
> > > > up the detection stuff, but it won't actually fix anything yet.  I
> > > > want to make sure I've got detection part right before I go messing
> > > > with the file system.  If you can pull and build and then run
> > > >
> > > > btrfs rescue recover-chunks <device>
> > > >
> > > > and capture the output that would be great.  Hopefully this shows the
> > > > missing block groups and I can just copy them into place.  Thanks,
> > >
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> > > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > > Csum didn't match
> > > ERROR: cannot read chunk root
> > > WTF???
> > > ERROR: open ctree failed, try btrfs rescue tree-recover
> > > Recover chunks tree failed
> > > gargamel:/var/local/src/btrfs-progs-josefbacik#
> > >
> > > So, should I do what it says?
> >
> > Oh yeah duh sorry, I was adding it to tree-recover but decided it was
> > a bad place for it.  Go ahead and run tree-recover and then do the
> > recover-chunks.  Thanks,
>
> :(
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> ERROR: cannot read chunk root
> WTF???
> ERROR: open ctree failed
> Tree recover failed

Sorry, thought I fixed this before pushing yesterday, try again please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-29 15:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-05-29 15:33                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-05-29 17:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-29 15:33 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 29, 2022 at 11:00:35AM -0400, Josef Bacik wrote:
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > ERROR: cannot read chunk root
> > WTF???
> > ERROR: open ctree failed
> > Tree recover failed
> 
> Sorry, thought I fixed this before pushing yesterday, try again please.  Thanks,

Resynced but it's the same:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
WARNING: cannot read chunk root, continue anyway
ERROR: Failed to read root block
Tree recover failed
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
Csum didn't match
ERROR: cannot read chunk root
WTF???
ERROR: open ctree failed, try btrfs rescue tree-recover
Recover chunks tree failed
gargamel:/var/local/src/btrfs-progs-josefbacik# 

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-29 15:33                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-05-29 17:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-05-29 18:05                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-29 17:32 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 29, 2022 at 11:33 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 29, 2022 at 11:00:35AM -0400, Josef Bacik wrote:
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > > ERROR: cannot read chunk root
> > > WTF???
> > > ERROR: open ctree failed
> > > Tree recover failed
> >
> > Sorry, thought I fixed this before pushing yesterday, try again please.  Thanks,
>
> Resynced but it's the same:
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> WARNING: cannot read chunk root, continue anyway
> ERROR: Failed to read root block
> Tree recover failed

Oh huh, apparently I only scan for a root if we didn't find a good fit
in the beginning, not if I couldn't read any roots.  Fixed that up,
please try again.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-29 17:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-05-29 18:05                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-05-29 18:58                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-29 18:05 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 29, 2022 at 01:32:48PM -0400, Josef Bacik wrote:
> On Sun, May 29, 2022 at 11:33 AM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sun, May 29, 2022 at 11:00:35AM -0400, Josef Bacik wrote:
> > > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > > > ERROR: cannot read chunk root
> > > > WTF???
> > > > ERROR: open ctree failed
> > > > Tree recover failed
> > >
> > > Sorry, thought I fixed this before pushing yesterday, try again please.  Thanks,
> >
> > Resynced but it's the same:
> >
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > WARNING: cannot read chunk root, continue anyway
> > ERROR: Failed to read root block
> > Tree recover failed
> 
> Oh huh, apparently I only scan for a root if we didn't find a good fit
> in the beginning, not if I couldn't read any roots.  Fixed that up,
> please try again.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
WARNING: cannot read chunk root, continue anyway
none of our backups was sufficient, scanning for a root
ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
Tree recover failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-29 18:05                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-05-29 18:58                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-05-29 19:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-29 18:58 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 29, 2022 at 2:05 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 29, 2022 at 01:32:48PM -0400, Josef Bacik wrote:
> > On Sun, May 29, 2022 at 11:33 AM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sun, May 29, 2022 at 11:00:35AM -0400, Josef Bacik wrote:
> > > > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > > > > ERROR: cannot read chunk root
> > > > > WTF???
> > > > > ERROR: open ctree failed
> > > > > Tree recover failed
> > > >
> > > > Sorry, thought I fixed this before pushing yesterday, try again please.  Thanks,
> > >
> > > Resynced but it's the same:
> > >
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > > WARNING: cannot read chunk root, continue anyway
> > > ERROR: Failed to read root block
> > > Tree recover failed
> >
> > Oh huh, apparently I only scan for a root if we didn't find a good fit
> > in the beginning, not if I couldn't read any roots.  Fixed that up,
> > please try again.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> WARNING: cannot read chunk root, continue anyway
> none of our backups was sufficient, scanning for a root
> ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
> Tree recover failed
>

Well if anything tree-recover will be rock solid by the end of this.
Try again please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-29 18:58                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-05-29 19:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-05-29 19:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-29 19:42 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 29, 2022 at 02:58:11PM -0400, Josef Bacik wrote:
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > WARNING: cannot read chunk root, continue anyway
> > none of our backups was sufficient, scanning for a root
> > ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
> > Tree recover failed
> 
> Well if anything tree-recover will be rock solid by the end of this.
> Try again please.  Thanks,

Sorry :(

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
WARNING: cannot read chunk root, continue anyway
none of our backups was sufficient, scanning for a root
ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
Tree recover failed
gargamel:/var/local/src/btrfs-progs-josefbacik# git pull
Already up-to-date.
gargamel:/var/local/src/btrfs-progs-josefbacik#


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-29 19:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-05-29 19:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-05-29 20:04                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-29 19:49 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 29, 2022 at 3:42 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 29, 2022 at 02:58:11PM -0400, Josef Bacik wrote:
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > > WARNING: cannot read chunk root, continue anyway
> > > none of our backups was sufficient, scanning for a root
> > > ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
> > > Tree recover failed
> >
> > Well if anything tree-recover will be rock solid by the end of this.
> > Try again please.  Thanks,
>
> Sorry :(
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> WARNING: cannot read chunk root, continue anyway
> none of our backups was sufficient, scanning for a root
> ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
> Tree recover failed

Lord alright, lets try some debugging.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-29 19:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-05-29 20:04                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-05-29 20:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-29 20:04 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 29, 2022 at 03:49:26PM -0400, Josef Bacik wrote:
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > WARNING: cannot read chunk root, continue anyway
> > none of our backups was sufficient, scanning for a root
> > ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
> > Tree recover failed
> 
> Lord alright, lets try some debugging.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
WARNING: cannot read chunk root, continue anyway
none of our backups was sufficient, scanning for a root
scanning, best has 0 found 1 bad
ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
Tree recover failed
gargamel:/var/local/src/btrfs-progs-josefbacik# 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-29 20:04                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-05-29 20:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-05-30  0:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-29 20:32 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 29, 2022 at 4:04 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 29, 2022 at 03:49:26PM -0400, Josef Bacik wrote:
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > > WARNING: cannot read chunk root, continue anyway
> > > none of our backups was sufficient, scanning for a root
> > > ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
> > > Tree recover failed
> >
> > Lord alright, lets try some debugging.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> WARNING: cannot read chunk root, continue anyway
> none of our backups was sufficient, scanning for a root
> scanning, best has 0 found 1 bad
> ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
> Tree recover failed
> gargamel:/var/local/src/btrfs-progs-josefbacik#

Hmm, the sys array should be fine, try again, hopefully that'll clear
things up.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-29 20:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-05-30  0:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-05-30  1:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-30  0:37 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 29, 2022 at 04:32:19PM -0400, Josef Bacik wrote:
> On Sun, May 29, 2022 at 4:04 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sun, May 29, 2022 at 03:49:26PM -0400, Josef Bacik wrote:
> > > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > > > WARNING: cannot read chunk root, continue anyway
> > > > none of our backups was sufficient, scanning for a root
> > > > ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
> > > > Tree recover failed
> > >
> > > Lord alright, lets try some debugging.  Thanks,
> >
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > WARNING: cannot read chunk root, continue anyway
> > none of our backups was sufficient, scanning for a root
> > scanning, best has 0 found 1 bad
> > ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
> > Tree recover failed
> > gargamel:/var/local/src/btrfs-progs-josefbacik#
> 
> Hmm, the sys array should be fine, try again, hopefully that'll clear
> things up.  Thanks,

It went farther this time

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
WARNING: cannot read chunk root, continue anyway
none of our backups was sufficient, scanning for a root
scanning, best has 0 found 1 bad
ret is 0 offset 20971520 len 8388608
ret is -2 offset 20971520 len 8388608
checking block 22495232 generation 1572124 fs info generation 0
checking block 22462464 generation 1479229 fs info generation 0
checking block 22528000 generation 1572115 fs info generation 0
checking block 22446080 generation 1571791 fs info generation 0
checking block 22544384 generation 1556078 fs info generation 0
checking block 22511616 generation 1555799 fs info generation 0
checking block 22577152 generation 1586277 fs info generation 0
checking block 22478848 generation 1561557 fs info generation 0
checking block 22593536 generation 1590219 fs info generation 0
checking block 22609920 generation 1551635 fs info generation 0
checking block 22560768 generation 1590217 fs info generation 0
ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
Tree recover failed

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-30  0:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-05-30  1:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-05-30 19:18                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-30  1:14 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, May 29, 2022 at 8:37 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 29, 2022 at 04:32:19PM -0400, Josef Bacik wrote:
> > On Sun, May 29, 2022 at 4:04 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sun, May 29, 2022 at 03:49:26PM -0400, Josef Bacik wrote:
> > > > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > > > > WARNING: cannot read chunk root, continue anyway
> > > > > none of our backups was sufficient, scanning for a root
> > > > > ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
> > > > > Tree recover failed
> > > >
> > > > Lord alright, lets try some debugging.  Thanks,
> > >
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> > > WARNING: cannot read chunk root, continue anyway
> > > none of our backups was sufficient, scanning for a root
> > > scanning, best has 0 found 1 bad
> > > ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
> > > Tree recover failed
> > > gargamel:/var/local/src/btrfs-progs-josefbacik#
> >
> > Hmm, the sys array should be fine, try again, hopefully that'll clear
> > things up.  Thanks,
>
> It went farther this time
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> WARNING: cannot read chunk root, continue anyway
> none of our backups was sufficient, scanning for a root
> scanning, best has 0 found 1 bad
> ret is 0 offset 20971520 len 8388608
> ret is -2 offset 20971520 len 8388608
> checking block 22495232 generation 1572124 fs info generation 0
> checking block 22462464 generation 1479229 fs info generation 0
> checking block 22528000 generation 1572115 fs info generation 0
> checking block 22446080 generation 1571791 fs info generation 0
> checking block 22544384 generation 1556078 fs info generation 0
> checking block 22511616 generation 1555799 fs info generation 0
> checking block 22577152 generation 1586277 fs info generation 0
> checking block 22478848 generation 1561557 fs info generation 0
> checking block 22593536 generation 1590219 fs info generation 0
> checking block 22609920 generation 1551635 fs info generation 0
> checking block 22560768 generation 1590217 fs info generation 0
> ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
> Tree recover failed
>

Ah ok that makes sense, fixed it, sorry about that.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-30  1:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-05-30 19:18                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-05-30 20:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-30 19:18 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, May 29, 2022 at 09:14:23PM -0400, Josef Bacik wrote:
> Ah ok that makes sense, fixed it, sorry about that.  Thanks,

Same?

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
WARNING: cannot read chunk root, continue anyway
none of our backups was sufficient, scanning for a root
scanning, best has 0 found 1 bad
ret is 0 offset 20971520 len 8388608
ret is -2 offset 20971520 len 8388608
checking block 22495232 generation 1572124 fs info generation 0
checking block 22462464 generation 1479229 fs info generation 0
checking block 22528000 generation 1572115 fs info generation 0
checking block 22446080 generation 1571791 fs info generation 0
checking block 22544384 generation 1556078 fs info generation 0
checking block 22511616 generation 1555799 fs info generation 0
checking block 22577152 generation 1586277 fs info generation 0
checking block 22478848 generation 1561557 fs info generation 0
checking block 22593536 generation 1590219 fs info generation 0
checking block 22609920 generation 1551635 fs info generation 0
checking block 22560768 generation 1590217 fs info generation 0
ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best
Tree recover failed
gargamel:/var/local/src/btrfs-progs-josefbacik# git pull
Already up-to-date.

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-30 19:18                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-05-30 20:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-05-31  1:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-30 20:53 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 30, 2022 at 3:18 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, May 29, 2022 at 09:14:23PM -0400, Josef Bacik wrote:
> > Ah ok that makes sense, fixed it, sorry about that.  Thanks,
>
> Same?

Bah my bad, we fail earlier than I realized, should work now.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-30 20:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-05-31  1:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-05-31 20:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-31  1:12 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, May 30, 2022 at 04:53:13PM -0400, Josef Bacik wrote:
> On Mon, May 30, 2022 at 3:18 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sun, May 29, 2022 at 09:14:23PM -0400, Josef Bacik wrote:
> > > Ah ok that makes sense, fixed it, sorry about that.  Thanks,
> >
> > Same?
> 
> Bah my bad, we fail earlier than I realized, should work now.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
WARNING: cannot read chunk root, continue anyway
none of our backups was sufficient, scanning for a root
scanning, best has 0 found 1 bad
ret is 0 offset 20971520 len 8388608
ret is -2 offset 20971520 len 8388608
checking block 22495232 generation 1572124 fs info generation 2582703
trying bytenr 22495232 got 1 blocks 0 bad
checking block 22462464 generation 1479229 fs info generation 2582703
trying bytenr 22462464 got 1 blocks 0 bad
checking block 22528000 generation 1572115 fs info generation 2582703
trying bytenr 22528000 got 1 blocks 0 bad
checking block 22446080 generation 1571791 fs info generation 2582703
trying bytenr 22446080 got 1 blocks 0 bad
checking block 22544384 generation 1556078 fs info generation 2582703
trying bytenr 22544384 got 1 blocks 0 bad
checking block 22511616 generation 1555799 fs info generation 2582703
trying bytenr 22511616 got 1 blocks 0 bad
checking block 22577152 generation 1586277 fs info generation 2582703
trying bytenr 22577152 got 1 blocks 0 bad
checking block 22478848 generation 1561557 fs info generation 2582703
trying bytenr 22478848 got 1 blocks 0 bad
checking block 22593536 generation 1590219 fs info generation 2582703
trying bytenr 22593536 got 1 blocks 0 bad
checking block 22609920 generation 1551635 fs info generation 2582703
trying bytenr 22609920 got 1 blocks 0 bad
checking block 22560768 generation 1590217 fs info generation 2582703
trying bytenr 22560768 got 1 blocks 0 bad
No mapping for 15645202989056-15645203005440
Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
No mapping for 15645202907136-15645202923520
Couldn't map the block 15645202907136
Couldn't map the block 15645202907136
No mapping for 15645202989056-15645203005440
Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
No mapping for 15645202989056-15645203005440
Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
No mapping for 15645202907136-15645202923520
Couldn't map the block 15645202907136
Couldn't map the block 15645202907136
none of our backups was sufficient, scanning for a root
scanning, best has 0 found 1 bad
ERROR: Couldn't find a valid root block for 1, we're going to clear it and hope for the best
Tree recover failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-31  1:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-05-31 20:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-05-31 22:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-05-31 20:57 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, May 30, 2022 at 9:12 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, May 30, 2022 at 04:53:13PM -0400, Josef Bacik wrote:
> > On Mon, May 30, 2022 at 3:18 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sun, May 29, 2022 at 09:14:23PM -0400, Josef Bacik wrote:
> > > > Ah ok that makes sense, fixed it, sorry about that.  Thanks,
> > >
> > > Same?
> >
> > Bah my bad, we fail earlier than I realized, should work now.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> WARNING: cannot read chunk root, continue anyway
> none of our backups was sufficient, scanning for a root
> scanning, best has 0 found 1 bad
> ret is 0 offset 20971520 len 8388608
> ret is -2 offset 20971520 len 8388608
> checking block 22495232 generation 1572124 fs info generation 2582703
> trying bytenr 22495232 got 1 blocks 0 bad
> checking block 22462464 generation 1479229 fs info generation 2582703
> trying bytenr 22462464 got 1 blocks 0 bad
> checking block 22528000 generation 1572115 fs info generation 2582703
> trying bytenr 22528000 got 1 blocks 0 bad
> checking block 22446080 generation 1571791 fs info generation 2582703
> trying bytenr 22446080 got 1 blocks 0 bad
> checking block 22544384 generation 1556078 fs info generation 2582703
> trying bytenr 22544384 got 1 blocks 0 bad
> checking block 22511616 generation 1555799 fs info generation 2582703
> trying bytenr 22511616 got 1 blocks 0 bad
> checking block 22577152 generation 1586277 fs info generation 2582703
> trying bytenr 22577152 got 1 blocks 0 bad
> checking block 22478848 generation 1561557 fs info generation 2582703
> trying bytenr 22478848 got 1 blocks 0 bad
> checking block 22593536 generation 1590219 fs info generation 2582703
> trying bytenr 22593536 got 1 blocks 0 bad
> checking block 22609920 generation 1551635 fs info generation 2582703
> trying bytenr 22609920 got 1 blocks 0 bad
> checking block 22560768 generation 1590217 fs info generation 2582703
> trying bytenr 22560768 got 1 blocks 0 bad
> No mapping for 15645202989056-15645203005440
> Couldn't map the block 15645202989056
> Couldn't map the block 15645202989056
> No mapping for 15645202907136-15645202923520
> Couldn't map the block 15645202907136
> Couldn't map the block 15645202907136
> No mapping for 15645202989056-15645203005440
> Couldn't map the block 15645202989056
> Couldn't map the block 15645202989056
> No mapping for 15645202989056-15645203005440
> Couldn't map the block 15645202989056
> Couldn't map the block 15645202989056
> No mapping for 15645202907136-15645202923520
> Couldn't map the block 15645202907136
> Couldn't map the block 15645202907136
> none of our backups was sufficient, scanning for a root
> scanning, best has 0 found 1 bad
> ERROR: Couldn't find a valid root block for 1, we're going to clear it and hope for the best
> Tree recover failed

I hate myself so much.  It looks like it recovered the chunk tree, so
you should be able to run

btrfs rescue recover-chunks <device>

to see if it can find the mapping's we're missing.  If it does then
I'll wire up the code to insert them, and then we can go about finding
the other roots and getting this thing fixed.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-31 20:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-05-31 22:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-06-01  0:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-05-31 22:49 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, May 31, 2022 at 04:57:17PM -0400, Josef Bacik wrote:
> I hate myself so much.  It looks like it recovered the chunk tree, so
> you should be able to run
> 
> btrfs rescue recover-chunks <device>
> 
> to see if it can find the mapping's we're missing.  If it does then
> I'll wire up the code to insert them, and then we can go about finding
> the other roots and getting this thing fixed.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# git pull
Already up-to-date.

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
Csum didn't match
ERROR: cannot read chunk root
WTF???
ERROR: open ctree failed, try btrfs rescue tree-recover
Recover chunks tree failed

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
WARNING: cannot read chunk root, continue anyway
none of our backups was sufficient, scanning for a root
scanning, best has 0 found 1 bad
ret is 0 offset 20971520 len 8388608
ret is -2 offset 20971520 len 8388608
checking block 22495232 generation 1572124 fs info generation 2582703
trying bytenr 22495232 got 1 blocks 0 bad
checking block 22462464 generation 1479229 fs info generation 2582703
trying bytenr 22462464 got 1 blocks 0 bad
checking block 22528000 generation 1572115 fs info generation 2582703
trying bytenr 22528000 got 1 blocks 0 bad
checking block 22446080 generation 1571791 fs info generation 2582703
trying bytenr 22446080 got 1 blocks 0 bad
checking block 22544384 generation 1556078 fs info generation 2582703
trying bytenr 22544384 got 1 blocks 0 bad
checking block 22511616 generation 1555799 fs info generation 2582703
trying bytenr 22511616 got 1 blocks 0 bad
checking block 22577152 generation 1586277 fs info generation 2582703
trying bytenr 22577152 got 1 blocks 0 bad
checking block 22478848 generation 1561557 fs info generation 2582703
trying bytenr 22478848 got 1 blocks 0 bad
checking block 22593536 generation 1590219 fs info generation 2582703
trying bytenr 22593536 got 1 blocks 0 bad
checking block 22609920 generation 1551635 fs info generation 2582703
trying bytenr 22609920 got 1 blocks 0 bad
checking block 22560768 generation 1590217 fs info generation 2582703
trying bytenr 22560768 got 1 blocks 0 bad
No mapping for 15645202989056-15645203005440
Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
No mapping for 15645202907136-15645202923520
Couldn't map the block 15645202907136
Couldn't map the block 15645202907136
No mapping for 15645202989056-15645203005440
Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
No mapping for 15645202989056-15645203005440
Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
No mapping for 15645202907136-15645202923520
Couldn't map the block 15645202907136
Couldn't map the block 15645202907136
none of our backups was sufficient, scanning for a root
scanning, best has 0 found 1 bad
ERROR: Couldn't find a valid root block for 1, we're going to clear it and hope for the best
Tree recover failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-05-31 22:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-06-01  0:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-01  0:25                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-01  0:14 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, May 31, 2022 at 6:49 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, May 31, 2022 at 04:57:17PM -0400, Josef Bacik wrote:
> > I hate myself so much.  It looks like it recovered the chunk tree, so
> > you should be able to run
> >
> > btrfs rescue recover-chunks <device>
> >
> > to see if it can find the mapping's we're missing.  If it does then
> > I'll wire up the code to insert them, and then we can go about finding
> > the other roots and getting this thing fixed.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# git pull
> Already up-to-date.
>

Wtf, we're clearly writing the chunk root properly because I have to
re-open it to recover the tree root, and that's where it fails, but
then the chunk restore can't open the root, despite it being correctly
read in the tree recover.  I've pushed new code, try tree-recover and
then recover-chunks again and capture the output please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01  0:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-06-01  0:25                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-01  1:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-01  0:25 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, May 31, 2022 at 08:14:27PM -0400, Josef Bacik wrote:
> Wtf, we're clearly writing the chunk root properly because I have to
> re-open it to recover the tree root, and that's where it fails, but
> then the chunk restore can't open the root, despite it being correctly
> read in the tree recover.  I've pushed new code, try tree-recover and
> then recover-chunks again and capture the output please.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
Csum didn't match
ERROR: cannot read chunk root 21135360
WTF???
ERROR: open ctree failed, try btrfs rescue tree-recover
Recover chunks tree failed
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
WARNING: cannot read chunk root, continue anyway
none of our backups was sufficient, scanning for a root
scanning, best has 0 found 1 bad
ret is 0 offset 20971520 len 8388608
ret is -2 offset 20971520 len 8388608
checking block 22495232 generation 1572124 fs info generation 2582703
trying bytenr 22495232 got 1 blocks 0 bad
checking block 22462464 generation 1479229 fs info generation 2582703
trying bytenr 22462464 got 1 blocks 0 bad
checking block 22528000 generation 1572115 fs info generation 2582703
trying bytenr 22528000 got 1 blocks 0 bad
checking block 22446080 generation 1571791 fs info generation 2582703
trying bytenr 22446080 got 1 blocks 0 bad
checking block 22544384 generation 1556078 fs info generation 2582703
trying bytenr 22544384 got 1 blocks 0 bad
checking block 22511616 generation 1555799 fs info generation 2582703
trying bytenr 22511616 got 1 blocks 0 bad
checking block 22577152 generation 1586277 fs info generation 2582703
trying bytenr 22577152 got 1 blocks 0 bad
checking block 22478848 generation 1561557 fs info generation 2582703
trying bytenr 22478848 got 1 blocks 0 bad
checking block 22593536 generation 1590219 fs info generation 2582703
trying bytenr 22593536 got 1 blocks 0 bad
checking block 22609920 generation 1551635 fs info generation 2582703
trying bytenr 22609920 got 1 blocks 0 bad
checking block 22560768 generation 1590217 fs info generation 2582703
trying bytenr 22560768 got 1 blocks 0 bad
No mapping for 15645202989056-15645203005440
Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
No mapping for 15645202907136-15645202923520
Couldn't map the block 15645202907136
Couldn't map the block 15645202907136
No mapping for 15645202989056-15645203005440
Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
No mapping for 15645202989056-15645203005440
Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
No mapping for 15645202907136-15645202923520
Couldn't map the block 15645202907136
Couldn't map the block 15645202907136
none of our backups was sufficient, scanning for a root
scanning, best has 0 found 1 bad
ERROR: Couldn't find a valid root block for 1, we're going to clear it and hope for the best
Tree recover failed
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
Csum didn't match
ERROR: cannot read chunk root 21135360
WTF???
ERROR: open ctree failed, try btrfs rescue tree-recover
Recover chunks tree failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01  0:25                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-01  1:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-06-01  1:29                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-01  1:26 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, May 31, 2022 at 8:25 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, May 31, 2022 at 08:14:27PM -0400, Josef Bacik wrote:
> > Wtf, we're clearly writing the chunk root properly because I have to
> > re-open it to recover the tree root, and that's where it fails, but
> > then the chunk restore can't open the root, despite it being correctly
> > read in the tree recover.  I've pushed new code, try tree-recover and
> > then recover-chunks again and capture the output please.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5

Ah ok, I wasn't actually updating the pointer, fixed that, lets try
the same sequence again.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01  1:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-06-01  1:29                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-06-01  2:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-01  1:29 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, May 31, 2022 at 09:26:03PM -0400, Josef Bacik wrote:
> On Tue, May 31, 2022 at 8:25 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Tue, May 31, 2022 at 08:14:27PM -0400, Josef Bacik wrote:
> > > Wtf, we're clearly writing the chunk root properly because I have to
> > > re-open it to recover the tree root, and that's where it fails, but
> > > then the chunk restore can't open the root, despite it being correctly
> > > read in the tree recover.  I've pushed new code, try tree-recover and
> > > then recover-chunks again and capture the output please.  Thanks,
> >
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> 
> Ah ok, I wasn't actually updating the pointer, fixed that, lets try
> the same sequence again.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# gdb ./btrfs rescue tree-recover /dev/mapper/dshelf1
Excess command line arguments ignored. (tree-recover ...)
GNU gdb (Debian 7.12-6+b2) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./btrfs...done.
/var/local/src/btrfs-progs-josefbacik/rescue: No such file or directory.
(gdb) rn rescue tree-recover /dev/mapper/dshelf1
Target exec does not support this command.
(gdb) run rescue tree-recover /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue tree-recover /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
WARNING: cannot read chunk root, continue anyway
none of our backups was sufficient, scanning for a root
scanning, best has 0 found 1 bad
ret is 0 offset 20971520 len 8388608
ret is -2 offset 20971520 len 8388608
checking block 22495232 generation 1572124 fs info generation 2582703
trying bytenr 22495232 got 1 blocks 0 bad
checking block 22462464 generation 1479229 fs info generation 2582703
trying bytenr 22462464 got 1 blocks 0 bad
checking block 22528000 generation 1572115 fs info generation 2582703
trying bytenr 22528000 got 1 blocks 0 bad
checking block 22446080 generation 1571791 fs info generation 2582703
trying bytenr 22446080 got 1 blocks 0 bad
checking block 22544384 generation 1556078 fs info generation 2582703
trying bytenr 22544384 got 1 blocks 0 bad
checking block 22511616 generation 1555799 fs info generation 2582703
trying bytenr 22511616 got 1 blocks 0 bad
checking block 22577152 generation 1586277 fs info generation 2582703
trying bytenr 22577152 got 1 blocks 0 bad
checking block 22478848 generation 1561557 fs info generation 2582703
trying bytenr 22478848 got 1 blocks 0 bad
checking block 22593536 generation 1590219 fs info generation 2582703
trying bytenr 22593536 got 1 blocks 0 bad
checking block 22609920 generation 1551635 fs info generation 2582703
trying bytenr 22609920 got 1 blocks 0 bad
checking block 22560768 generation 1590217 fs info generation 2582703
trying bytenr 22560768 got 1 blocks 0 bad
ret is 0 offset 20971520 len 8388608
ret is -2 offset 20971520 len 8388608
setting chunk root to 22593536

Program received signal SIGSEGV, Segmentation fault.
0x000055555557b352 in backup_super_roots (info=0x55555564fbc0) at ./kernel-shared/ctree.h:2270
2270    BTRFS_SETGET_STACK_FUNCS(backup_tree_root_gen, struct btrfs_root_backup,
(gdb) bt
#0  0x000055555557b352 in backup_super_roots (info=0x55555564fbc0) at ./kernel-shared/ctree.h:2270
#1  write_all_supers (fs_info=fs_info@entry=0x55555564fbc0) at kernel-shared/disk-io.c:2102
#2  0x00005555555df49e in repair_super_root (fs_info_ptr=fs_info_ptr@entry=0x7fffffffdce8, ocf=ocf@entry=0x7fffffffdcc0, 
    objectid=objectid@entry=3) at cmds/rescue-tree-recover.c:958
#3  0x00005555555df62c in btrfs_recover_trees (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
    at cmds/rescue-tree-recover.c:1194
#4  0x00005555555d808e in cmd_rescue_tree_recover (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:176
#5  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645c20 <cmd_struct_rescue_tree_recover>)
    at cmds/commands.h:125
#6  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01  1:29                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-06-01  2:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-06-01  3:15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-01  2:10 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, May 31, 2022 at 9:29 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, May 31, 2022 at 09:26:03PM -0400, Josef Bacik wrote:
> > On Tue, May 31, 2022 at 8:25 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Tue, May 31, 2022 at 08:14:27PM -0400, Josef Bacik wrote:
> > > > Wtf, we're clearly writing the chunk root properly because I have to
> > > > re-open it to recover the tree root, and that's where it fails, but
> > > > then the chunk restore can't open the root, despite it being correctly
> > > > read in the tree recover.  I've pushed new code, try tree-recover and
> > > > then recover-chunks again and capture the output please.  Thanks,
> > >
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> > > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> >
> > Ah ok, I wasn't actually updating the pointer, fixed that, lets try
> > the same sequence again.  Thanks,
>

Ok backup roots don't work if we didn't read them, try again please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01  2:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-06-01  3:15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-06-01 13:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-01  3:15 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, May 31, 2022 at 10:10:49PM -0400, Josef Bacik wrote:
> On Tue, May 31, 2022 at 9:29 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Tue, May 31, 2022 at 09:26:03PM -0400, Josef Bacik wrote:
> > > On Tue, May 31, 2022 at 8:25 PM Marc MERLIN <marc@merlins.org> wrote:
> > > >
> > > > On Tue, May 31, 2022 at 08:14:27PM -0400, Josef Bacik wrote:
> > > > > Wtf, we're clearly writing the chunk root properly because I have to
> > > > > re-open it to recover the tree root, and that's where it fails, but
> > > > > then the chunk restore can't open the root, despite it being correctly
> > > > > read in the tree recover.  I've pushed new code, try tree-recover and
> > > > > then recover-chunks again and capture the output please.  Thanks,
> > > >
> > > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> > > > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > > > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > > > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > >
> > > Ah ok, I wasn't actually updating the pointer, fixed that, lets try
> > > the same sequence again.  Thanks,
> >
> 
> Ok backup roots don't work if we didn't read them, try again please.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# git pull
Already up-to-date.
gargamel:/var/local/src/btrfs-progs-josefbacik# gdb ./btrfs rescue tree-recover /dev/mapper/dshelf1
Excess command line arguments ignored. (tree-recover ...)
GNU gdb (Debian 7.12-6+b2) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./btrfs...done.
/var/local/src/btrfs-progs-josefbacik/rescue: No such file or directory.
(gdb) run rescue tree-recover /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue tree-recover /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
WARNING: cannot read chunk root, continue anyway
none of our backups was sufficient, scanning for a root
scanning, best has 0 found 1 bad
ret is 0 offset 20971520 len 8388608
ret is -2 offset 20971520 len 8388608
checking block 22495232 generation 1572124 fs info generation 2582703
trying bytenr 22495232 got 1 blocks 0 bad
checking block 22462464 generation 1479229 fs info generation 2582703
trying bytenr 22462464 got 1 blocks 0 bad
checking block 22528000 generation 1572115 fs info generation 2582703
trying bytenr 22528000 got 1 blocks 0 bad
checking block 22446080 generation 1571791 fs info generation 2582703
trying bytenr 22446080 got 1 blocks 0 bad
checking block 22544384 generation 1556078 fs info generation 2582703
trying bytenr 22544384 got 1 blocks 0 bad
checking block 22511616 generation 1555799 fs info generation 2582703
trying bytenr 22511616 got 1 blocks 0 bad
checking block 22577152 generation 1586277 fs info generation 2582703
trying bytenr 22577152 got 1 blocks 0 bad
checking block 22478848 generation 1561557 fs info generation 2582703
trying bytenr 22478848 got 1 blocks 0 bad
checking block 22593536 generation 1590219 fs info generation 2582703
trying bytenr 22593536 got 1 blocks 0 bad
checking block 22609920 generation 1551635 fs info generation 2582703
trying bytenr 22609920 got 1 blocks 0 bad
checking block 22560768 generation 1590217 fs info generation 2582703
trying bytenr 22560768 got 1 blocks 0 bad
ret is 0 offset 20971520 len 8388608
ret is -2 offset 20971520 len 8388608
setting chunk root to 22593536

Program received signal SIGSEGV, Segmentation fault.
0x000055555557b328 in backup_super_roots (info=0x55555564fbc0) at kernel-shared/disk-io.c:2027
2027            if (!info->chunk_root->node ||
(gdb) bt
#0  0x000055555557b328 in backup_super_roots (info=0x55555564fbc0) at kernel-shared/disk-io.c:2027
#1  write_all_supers (fs_info=fs_info@entry=0x55555564fbc0) at kernel-shared/disk-io.c:2110
#2  0x00005555555df4ef in repair_super_root (fs_info_ptr=fs_info_ptr@entry=0x7fffffffdce8, ocf=ocf@entry=0x7fffffffdcc0, 
    objectid=objectid@entry=3) at cmds/rescue-tree-recover.c:958
#3  0x00005555555df67d in btrfs_recover_trees (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
    at cmds/rescue-tree-recover.c:1194
#4  0x00005555555d80df in cmd_rescue_tree_recover (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:176
#5  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645c20 <cmd_struct_rescue_tree_recover>)
    at cmds/commands.h:125
#6  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01  3:15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-06-01 13:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-06-01 16:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-01 13:56 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, May 31, 2022 at 11:15 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, May 31, 2022 at 10:10:49PM -0400, Josef Bacik wrote:
> > On Tue, May 31, 2022 at 9:29 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Tue, May 31, 2022 at 09:26:03PM -0400, Josef Bacik wrote:
> > > > On Tue, May 31, 2022 at 8:25 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > >
> > > > > On Tue, May 31, 2022 at 08:14:27PM -0400, Josef Bacik wrote:
> > > > > > Wtf, we're clearly writing the chunk root properly because I have to
> > > > > > re-open it to recover the tree root, and that's where it fails, but
> > > > > > then the chunk restore can't open the root, despite it being correctly
> > > > > > read in the tree recover.  I've pushed new code, try tree-recover and
> > > > > > then recover-chunks again and capture the output please.  Thanks,
> > > > >
> > > > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> > > > > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > > > > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > > > > checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
> > > >
> > > > Ah ok, I wasn't actually updating the pointer, fixed that, lets try
> > > > the same sequence again.  Thanks,
> > >
> >
> > Ok backup roots don't work if we didn't read them, try again please.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# git pull
> Already up-to-date.
> gargamel:/var/local/src/btrfs-progs-josefbacik# gdb ./btrfs rescue tree-recover /dev/mapper/dshelf1
> Excess command line arguments ignored. (tree-recover ...)
> GNU gdb (Debian 7.12-6+b2) 7.12.0.20161007-git
> Copyright (C) 2016 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from ./btrfs...done.
> /var/local/src/btrfs-progs-josefbacik/rescue: No such file or directory.
> (gdb) run rescue tree-recover /dev/mapper/dshelf1
> Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue tree-recover /dev/mapper/dshelf1
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> WARNING: cannot read chunk root, continue anyway
> none of our backups was sufficient, scanning for a root
> scanning, best has 0 found 1 bad
> ret is 0 offset 20971520 len 8388608
> ret is -2 offset 20971520 len 8388608
> checking block 22495232 generation 1572124 fs info generation 2582703
> trying bytenr 22495232 got 1 blocks 0 bad
> checking block 22462464 generation 1479229 fs info generation 2582703
> trying bytenr 22462464 got 1 blocks 0 bad
> checking block 22528000 generation 1572115 fs info generation 2582703
> trying bytenr 22528000 got 1 blocks 0 bad
> checking block 22446080 generation 1571791 fs info generation 2582703
> trying bytenr 22446080 got 1 blocks 0 bad
> checking block 22544384 generation 1556078 fs info generation 2582703
> trying bytenr 22544384 got 1 blocks 0 bad
> checking block 22511616 generation 1555799 fs info generation 2582703
> trying bytenr 22511616 got 1 blocks 0 bad
> checking block 22577152 generation 1586277 fs info generation 2582703
> trying bytenr 22577152 got 1 blocks 0 bad
> checking block 22478848 generation 1561557 fs info generation 2582703
> trying bytenr 22478848 got 1 blocks 0 bad
> checking block 22593536 generation 1590219 fs info generation 2582703
> trying bytenr 22593536 got 1 blocks 0 bad
> checking block 22609920 generation 1551635 fs info generation 2582703
> trying bytenr 22609920 got 1 blocks 0 bad
> checking block 22560768 generation 1590217 fs info generation 2582703
> trying bytenr 22560768 got 1 blocks 0 bad
> ret is 0 offset 20971520 len 8388608
> ret is -2 offset 20971520 len 8388608
> setting chunk root to 22593536
>

Sigh, try again please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01 13:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-06-01 16:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-06-01 18:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-01 16:39 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 01, 2022 at 09:56:14AM -0400, Josef Bacik wrote:
> Sigh, try again please.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
WARNING: cannot read chunk root, continue anyway
none of our backups was sufficient, scanning for a root
scanning, best has 0 found 1 bad
ret is 0 offset 20971520 len 8388608
ret is -2 offset 20971520 len 8388608
checking block 22495232 generation 1572124 fs info generation 2582703
trying bytenr 22495232 got 1 blocks 0 bad
checking block 22462464 generation 1479229 fs info generation 2582703
trying bytenr 22462464 got 1 blocks 0 bad
checking block 22528000 generation 1572115 fs info generation 2582703
trying bytenr 22528000 got 1 blocks 0 bad
checking block 22446080 generation 1571791 fs info generation 2582703
trying bytenr 22446080 got 1 blocks 0 bad
checking block 22544384 generation 1556078 fs info generation 2582703
trying bytenr 22544384 got 1 blocks 0 bad
checking block 22511616 generation 1555799 fs info generation 2582703
trying bytenr 22511616 got 1 blocks 0 bad
checking block 22577152 generation 1586277 fs info generation 2582703
trying bytenr 22577152 got 1 blocks 0 bad
checking block 22478848 generation 1561557 fs info generation 2582703
trying bytenr 22478848 got 1 blocks 0 bad
checking block 22593536 generation 1590219 fs info generation 2582703
trying bytenr 22593536 got 1 blocks 0 bad
checking block 22609920 generation 1551635 fs info generation 2582703
trying bytenr 22609920 got 1 blocks 0 bad
checking block 22560768 generation 1590217 fs info generation 2582703
trying bytenr 22560768 got 1 blocks 0 bad
ret is 0 offset 20971520 len 8388608
ret is -2 offset 20971520 len 8388608
setting chunk root to 22593536
FS_INFO IS 0x56263385f400
Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808
Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
Couldn't read tree root
FS_INFO AFTER IS 0x56263385f400
Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808
Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
Invalid mapping for 15645202907136-15645202923520, got 15664345513984-15665419255808
Couldn't map the block 15645202907136
Couldn't map the block 15645202907136
Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808
Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808
Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
Invalid mapping for 15645202907136-15645202923520, got 15664345513984-15665419255808
Couldn't map the block 15645202907136
Couldn't map the block 15645202907136
none of our backups was sufficient, scanning for a root
scanning, best has 0 found 1 bad
ERROR: Couldn't find a valid root block for 1, we're going to clear it and hope for the best
Tree recover failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01 16:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-06-01 18:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-01 18:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-01 18:00 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 1, 2022 at 12:39 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Jun 01, 2022 at 09:56:14AM -0400, Josef Bacik wrote:
> > Sigh, try again please.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> WARNING: cannot read chunk root, continue anyway
> none of our backups was sufficient, scanning for a root
> scanning, best has 0 found 1 bad
> ret is 0 offset 20971520 len 8388608
> ret is -2 offset 20971520 len 8388608
> checking block 22495232 generation 1572124 fs info generation 2582703
> trying bytenr 22495232 got 1 blocks 0 bad
> checking block 22462464 generation 1479229 fs info generation 2582703
> trying bytenr 22462464 got 1 blocks 0 bad
> checking block 22528000 generation 1572115 fs info generation 2582703
> trying bytenr 22528000 got 1 blocks 0 bad
> checking block 22446080 generation 1571791 fs info generation 2582703
> trying bytenr 22446080 got 1 blocks 0 bad
> checking block 22544384 generation 1556078 fs info generation 2582703
> trying bytenr 22544384 got 1 blocks 0 bad
> checking block 22511616 generation 1555799 fs info generation 2582703
> trying bytenr 22511616 got 1 blocks 0 bad
> checking block 22577152 generation 1586277 fs info generation 2582703
> trying bytenr 22577152 got 1 blocks 0 bad
> checking block 22478848 generation 1561557 fs info generation 2582703
> trying bytenr 22478848 got 1 blocks 0 bad
> checking block 22593536 generation 1590219 fs info generation 2582703
> trying bytenr 22593536 got 1 blocks 0 bad
> checking block 22609920 generation 1551635 fs info generation 2582703
> trying bytenr 22609920 got 1 blocks 0 bad
> checking block 22560768 generation 1590217 fs info generation 2582703
> trying bytenr 22560768 got 1 blocks 0 bad
> ret is 0 offset 20971520 len 8388608
> ret is -2 offset 20971520 len 8388608
> setting chunk root to 22593536

Ok perfect, now try btrfs rescue recover-chunks <device>, thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01 18:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-06-01 18:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-01 18:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-01 18:08 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 01, 2022 at 02:00:43PM -0400, Josef Bacik wrote:
> Ok perfect, now try btrfs rescue recover-chunks <device>, thanks,

(gdb) run rescue recover-chunks /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue recover-chunks /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564fbc0
Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808
Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0
Couldn't read tree root
FS_INFO AFTER IS 0x55555564fbc0
Walking all our trees and pinning down the currently accessible blocks

Program received signal SIGSEGV, Segmentation fault.
traverse_tree_blocks (tree=tree@entry=0x55555565a130, eb=0x0, tree_root=tree_root@entry=1) at common/repair.c:95
95              struct btrfs_fs_info *fs_info = eb->fs_info;
(gdb) bt
#0  traverse_tree_blocks (tree=tree@entry=0x55555565a130, eb=0x0, tree_root=tree_root@entry=1) at common/repair.c:95
#1  0x000055555559f6d4 in btrfs_mark_used_tree_blocks (fs_info=fs_info@entry=0x55555564fbc0, tree=tree@entry=0x55555565a130)
    at common/repair.c:188
#2  0x00005555555e2e18 in btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
    at cmds/rescue-recover-chunks.c:262
#3  0x00005555555d7e81 in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#4  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645d40 <cmd_struct_rescue_recover_chunks>)
    at cmds/commands.h:125
#5  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#6  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#7  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01 18:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-01 18:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-06-01 18:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-01 18:42 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 1, 2022 at 2:08 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Jun 01, 2022 at 02:00:43PM -0400, Josef Bacik wrote:
> > Ok perfect, now try btrfs rescue recover-chunks <device>, thanks,
>
> (gdb) run rescue recover-chunks /dev/mapper/dshelf1
> Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue recover-chunks /dev/mapper/dshelf1
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> FS_INFO IS 0x55555564fbc0
> Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808
> Couldn't map the block 15645202989056
> Couldn't map the block 15645202989056
> bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0
> Couldn't read tree root
> FS_INFO AFTER IS 0x55555564fbc0
> Walking all our trees and pinning down the currently accessible blocks

Ok you're ready to go.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01 18:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-06-01 18:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-06-01 19:01                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-01 18:50 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 01, 2022 at 02:42:55PM -0400, Josef Bacik wrote:
> On Wed, Jun 1, 2022 at 2:08 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Wed, Jun 01, 2022 at 02:00:43PM -0400, Josef Bacik wrote:
> > > Ok perfect, now try btrfs rescue recover-chunks <device>, thanks,
> >
> > (gdb) run rescue recover-chunks /dev/mapper/dshelf1
> > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue recover-chunks /dev/mapper/dshelf1
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > FS_INFO IS 0x55555564fbc0
> > Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808
> > Couldn't map the block 15645202989056
> > Couldn't map the block 15645202989056
> > bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0
> > Couldn't read tree root
> > FS_INFO AFTER IS 0x55555564fbc0
> > Walking all our trees and pinning down the currently accessible blocks
> 
> Ok you're ready to go.  Thanks,

Indeed, good job:

FS_INFO IS 0x55555564fbc0
Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808
Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0
Couldn't read tree root
FS_INFO AFTER IS 0x55555564fbc0
Walking all our trees and pinning down the currently accessible blocks
we would add a chunk at 14823605665792-14824679407616 type 0
we would add a chunk at 14824679407616-14825753149440 type 0
we would add a chunk at 14825753149440-14826826891264 type 0
(...)
we would add a chunk at 15772793438208-15773867180032 type 0
we would add a chunk at 15773867180032-15774940921856 type 0
we would add a chunk at 15774940921856-15776014663680 type 0
we would add a chunk at 15776014663680-15777088405504 type 0
we would add a chunk at 15777088405504-15778162147328 type 0
doing close???
Recover chunks succeeded, you can run check now
[Inferior 1 (process 696) exited normally]

Which btrfs check do you want me to run?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01 18:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-06-01 19:01                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-06-01 20:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-01 19:01 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 1, 2022 at 2:50 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Jun 01, 2022 at 02:42:55PM -0400, Josef Bacik wrote:
> > On Wed, Jun 1, 2022 at 2:08 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Wed, Jun 01, 2022 at 02:00:43PM -0400, Josef Bacik wrote:
> > > > Ok perfect, now try btrfs rescue recover-chunks <device>, thanks,
> > >
> > > (gdb) run rescue recover-chunks /dev/mapper/dshelf1
> > > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue recover-chunks /dev/mapper/dshelf1
> > > [Thread debugging using libthread_db enabled]
> > > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > > FS_INFO IS 0x55555564fbc0
> > > Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808
> > > Couldn't map the block 15645202989056
> > > Couldn't map the block 15645202989056
> > > bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0
> > > Couldn't read tree root
> > > FS_INFO AFTER IS 0x55555564fbc0
> > > Walking all our trees and pinning down the currently accessible blocks
> >
> > Ok you're ready to go.  Thanks,
>
> Indeed, good job:
>
> FS_INFO IS 0x55555564fbc0
> Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808
> Couldn't map the block 15645202989056
> Couldn't map the block 15645202989056
> bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0
> Couldn't read tree root
> FS_INFO AFTER IS 0x55555564fbc0
> Walking all our trees and pinning down the currently accessible blocks
> we would add a chunk at 14823605665792-14824679407616 type 0
> we would add a chunk at 14824679407616-14825753149440 type 0
> we would add a chunk at 14825753149440-14826826891264 type 0
> (...)
> we would add a chunk at 15772793438208-15773867180032 type 0
> we would add a chunk at 15773867180032-15774940921856 type 0
> we would add a chunk at 15774940921856-15776014663680 type 0
> we would add a chunk at 15776014663680-15777088405504 type 0
> we would add a chunk at 15777088405504-15778162147328 type 0
> doing close???
> Recover chunks succeeded, you can run check now
> [Inferior 1 (process 696) exited normally]
>
> Which btrfs check do you want me to run?
>

Phew ok, I'm finding the right chunks, that's perfect.  I'm now going
to write the code to fill in the missing chunks, give me a bit to do
that.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01 19:01                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-06-01 20:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  2022-06-01 21:40                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-01 20:57 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 1, 2022 at 3:01 PM Josef Bacik <josef@toxicpanda.com> wrote:
>
> On Wed, Jun 1, 2022 at 2:50 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Wed, Jun 01, 2022 at 02:42:55PM -0400, Josef Bacik wrote:
> > > On Wed, Jun 1, 2022 at 2:08 PM Marc MERLIN <marc@merlins.org> wrote:
> > > >
> > > > On Wed, Jun 01, 2022 at 02:00:43PM -0400, Josef Bacik wrote:
> > > > > Ok perfect, now try btrfs rescue recover-chunks <device>, thanks,
> > > >
> > > > (gdb) run rescue recover-chunks /dev/mapper/dshelf1
> > > > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue recover-chunks /dev/mapper/dshelf1
> > > > [Thread debugging using libthread_db enabled]
> > > > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > > > FS_INFO IS 0x55555564fbc0
> > > > Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808
> > > > Couldn't map the block 15645202989056
> > > > Couldn't map the block 15645202989056
> > > > bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0
> > > > Couldn't read tree root
> > > > FS_INFO AFTER IS 0x55555564fbc0
> > > > Walking all our trees and pinning down the currently accessible blocks
> > >
> > > Ok you're ready to go.  Thanks,
> >
> > Indeed, good job:
> >
> > FS_INFO IS 0x55555564fbc0
> > Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808
> > Couldn't map the block 15645202989056
> > Couldn't map the block 15645202989056
> > bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0
> > Couldn't read tree root
> > FS_INFO AFTER IS 0x55555564fbc0
> > Walking all our trees and pinning down the currently accessible blocks
> > we would add a chunk at 14823605665792-14824679407616 type 0
> > we would add a chunk at 14824679407616-14825753149440 type 0
> > we would add a chunk at 14825753149440-14826826891264 type 0
> > (...)
> > we would add a chunk at 15772793438208-15773867180032 type 0
> > we would add a chunk at 15773867180032-15774940921856 type 0
> > we would add a chunk at 15774940921856-15776014663680 type 0
> > we would add a chunk at 15776014663680-15777088405504 type 0
> > we would add a chunk at 15777088405504-15778162147328 type 0
> > doing close???
> > Recover chunks succeeded, you can run check now
> > [Inferior 1 (process 696) exited normally]
> >
> > Which btrfs check do you want me to run?
> >
>
> Phew ok, I'm finding the right chunks, that's perfect.  I'm now going
> to write the code to fill in the missing chunks, give me a bit to do
> that.  Thanks,
>

Ok I've committed the code, but I forsee all sorts of wonky problems
since we don't have a tree root yet, there may be a variety of
segfaults I have to run down before it actually works.  So go ahead
and do

btrfs rescue recover-chunks <device>

if by some miracle it completes, you'll then want to run

btrfs rescue tree-recover <device>

and then we can go from there.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01 20:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
@ 2022-06-01 21:40                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  2022-06-01 22:34                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-01 21:40 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 01, 2022 at 04:57:34PM -0400, Josef Bacik wrote:
> Ok I've committed the code, but I forsee all sorts of wonky problems
> since we don't have a tree root yet, there may be a variety of
> segfaults I have to run down before it actually works.  So go ahead
> and do
> 
> btrfs rescue recover-chunks <device>
> 
> if by some miracle it completes, you'll then want to run
 
Found missing chunk 15483956887552-15485030629376 type 0
Found missing chunk 15485030629376-15486104371200 type 0
Found missing chunk 15486104371200-15487178113024 type 0
Found missing chunk 15487178113024-15488251854848 type 0
Found missing chunk 15488251854848-15489325596672 type 0
Found missing chunk 15671861706752-15672935448576 type 0
Found missing chunk 15672935448576-15674009190400 type 0
Found missing chunk 15772793438208-15773867180032 type 0
Found missing chunk 15773867180032-15774940921856 type 0
Found missing chunk 15774940921856-15776014663680 type 0
Found missing chunk 15776014663680-15777088405504 type 0
Found missing chunk 15777088405504-15778162147328 type 0
ERROR: Corrupted fs, no valid METADATA block group found
Inserting chunk 14823605665792wtf transid 2582704

Program received signal SIGSEGV, Segmentation fault.
btrfs_print_leaf (eb=0x0, mode=mode@entry=0) at kernel-shared/print-tree.c:1274
1274            u32 leaf_data_size = __BTRFS_LEAF_DATA_SIZE(eb->len);
(gdb) bt
#0  btrfs_print_leaf (eb=0x0, mode=mode@entry=0) at kernel-shared/print-tree.c:1274
#1  0x000055555557730d in btrfs_insert_item (trans=trans@entry=0x555555662000, root=<optimized out>, 
    cpu_key=cpu_key@entry=0x55555564fb70, data=0x55555565de80, data_size=80) at kernel-shared/ctree.c:2931
#2  0x00005555555e2f88 in restore_missing_chunks (fs_info=0x55555564fbc0) at ./kernel-shared/ctree.h:322
#3  btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-recover-chunks.c:424
#4  0x00005555555d7c0f in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#5  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645d40 <cmd_struct_rescue_recover_chunks>)
    at cmds/commands.h:125
#6  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405


> btrfs rescue tree-recover <device>
> 
> and then we can go from there.  Thanks,
> 
> Josef
> 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01 21:40                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
@ 2022-06-01 22:34                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  2022-06-01 22:36                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-01 22:34 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 1, 2022 at 5:40 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Jun 01, 2022 at 04:57:34PM -0400, Josef Bacik wrote:
> > Ok I've committed the code, but I forsee all sorts of wonky problems
> > since we don't have a tree root yet, there may be a variety of
> > segfaults I have to run down before it actually works.  So go ahead
> > and do
> >
> > btrfs rescue recover-chunks <device>
> >
> > if by some miracle it completes, you'll then want to run
>
> Found missing chunk 15483956887552-15485030629376 type 0
> Found missing chunk 15485030629376-15486104371200 type 0
> Found missing chunk 15486104371200-15487178113024 type 0
> Found missing chunk 15487178113024-15488251854848 type 0
> Found missing chunk 15488251854848-15489325596672 type 0
> Found missing chunk 15671861706752-15672935448576 type 0
> Found missing chunk 15672935448576-15674009190400 type 0
> Found missing chunk 15772793438208-15773867180032 type 0
> Found missing chunk 15773867180032-15774940921856 type 0
> Found missing chunk 15774940921856-15776014663680 type 0
> Found missing chunk 15776014663680-15777088405504 type 0
> Found missing chunk 15777088405504-15778162147328 type 0
> ERROR: Corrupted fs, no valid METADATA block group found
> Inserting chunk 14823605665792wtf transid 2582704
>

Fixed, lets try that again please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01 22:34                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
@ 2022-06-01 22:36                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  2022-06-01 22:54                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-01 22:36 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 01, 2022 at 06:34:51PM -0400, Josef Bacik wrote:
> On Wed, Jun 1, 2022 at 5:40 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Wed, Jun 01, 2022 at 04:57:34PM -0400, Josef Bacik wrote:
> > > Ok I've committed the code, but I forsee all sorts of wonky problems
> > > since we don't have a tree root yet, there may be a variety of
> > > segfaults I have to run down before it actually works.  So go ahead
> > > and do
> > >
> > > btrfs rescue recover-chunks <device>
> > >
> > > if by some miracle it completes, you'll then want to run
> >
> > Found missing chunk 15483956887552-15485030629376 type 0
> > Found missing chunk 15485030629376-15486104371200 type 0
> > Found missing chunk 15486104371200-15487178113024 type 0
> > Found missing chunk 15487178113024-15488251854848 type 0
> > Found missing chunk 15488251854848-15489325596672 type 0
> > Found missing chunk 15671861706752-15672935448576 type 0
> > Found missing chunk 15672935448576-15674009190400 type 0
> > Found missing chunk 15772793438208-15773867180032 type 0
> > Found missing chunk 15773867180032-15774940921856 type 0
> > Found missing chunk 15774940921856-15776014663680 type 0
> > Found missing chunk 15776014663680-15777088405504 type 0
> > Found missing chunk 15777088405504-15778162147328 type 0
> > ERROR: Corrupted fs, no valid METADATA block group found
> > Inserting chunk 14823605665792wtf transid 2582704
> 
> Fixed, lets try that again please.  Thanks,

Found missing chunk 15672935448576-15674009190400 type 0
Found missing chunk 15772793438208-15773867180032 type 0
Found missing chunk 15773867180032-15774940921856 type 0
Found missing chunk 15774940921856-15776014663680 type 0
Found missing chunk 15776014663680-15777088405504 type 0
Found missing chunk 15777088405504-15778162147328 type 0
 Unable to find block group for 0
Unable to find block group for 0
Unable to find block group for 0
Inserting chunk 14823605665792wtf transid 2582704

Program received signal SIGSEGV, Segmentation fault.
btrfs_print_leaf (eb=0x0, mode=mode@entry=0) at kernel-shared/print-tree.c:1274
1274            u32 leaf_data_size = __BTRFS_LEAF_DATA_SIZE(eb->len);
(gdb)  bt
#0  btrfs_print_leaf (eb=0x0, mode=mode@entry=0) at kernel-shared/print-tree.c:1274
#1  0x000055555557730d in btrfs_insert_item (trans=trans@entry=0x555555662000, root=<optimized out>, 
    cpu_key=cpu_key@entry=0x55555564fb70, data=0x55555565de80, data_size=80) at kernel-shared/ctree.c:2931
#2  0x00005555555e2fb3 in restore_missing_chunks (fs_info=0x55555564fbc0) at ./kernel-shared/ctree.h:322
#3  btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-recover-chunks.c:424
#4  0x00005555555d7c3a in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#5  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645d40 <cmd_struct_rescue_recover_chunks>)
    at cmds/commands.h:125
#6  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01 22:36                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
@ 2022-06-01 22:54                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  2022-06-01 22:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-01 22:54 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 1, 2022 at 6:36 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Jun 01, 2022 at 06:34:51PM -0400, Josef Bacik wrote:
> > On Wed, Jun 1, 2022 at 5:40 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Wed, Jun 01, 2022 at 04:57:34PM -0400, Josef Bacik wrote:
> > > > Ok I've committed the code, but I forsee all sorts of wonky problems
> > > > since we don't have a tree root yet, there may be a variety of
> > > > segfaults I have to run down before it actually works.  So go ahead
> > > > and do
> > > >
> > > > btrfs rescue recover-chunks <device>
> > > >
> > > > if by some miracle it completes, you'll then want to run
> > >
> > > Found missing chunk 15483956887552-15485030629376 type 0
> > > Found missing chunk 15485030629376-15486104371200 type 0
> > > Found missing chunk 15486104371200-15487178113024 type 0
> > > Found missing chunk 15487178113024-15488251854848 type 0
> > > Found missing chunk 15488251854848-15489325596672 type 0
> > > Found missing chunk 15671861706752-15672935448576 type 0
> > > Found missing chunk 15672935448576-15674009190400 type 0
> > > Found missing chunk 15772793438208-15773867180032 type 0
> > > Found missing chunk 15773867180032-15774940921856 type 0
> > > Found missing chunk 15774940921856-15776014663680 type 0
> > > Found missing chunk 15776014663680-15777088405504 type 0
> > > Found missing chunk 15777088405504-15778162147328 type 0
> > > ERROR: Corrupted fs, no valid METADATA block group found
> > > Inserting chunk 14823605665792wtf transid 2582704
> >
> > Fixed, lets try that again please.  Thanks,
>
> Found missing chunk 15672935448576-15674009190400 type 0
> Found missing chunk 15772793438208-15773867180032 type 0
> Found missing chunk 15773867180032-15774940921856 type 0
> Found missing chunk 15774940921856-15776014663680 type 0
> Found missing chunk 15776014663680-15777088405504 type 0
> Found missing chunk 15777088405504-15778162147328 type 0
>  Unable to find block group for 0
> Unable to find block group for 0
> Unable to find block group for 0
> Inserting chunk 14823605665792wtf transid 2582704
>

Ah right, I have to mock up free space since we can't read our normal
stuff.  Fixed, lets go again.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01 22:54                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
@ 2022-06-01 22:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  2022-06-01 23:04                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-01 22:56 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 01, 2022 at 06:54:31PM -0400, Josef Bacik wrote:
> Ah right, I have to mock up free space since we can't read our normal
> stuff.  Fixed, lets go again.  Thanks,

Found missing chunk 15483956887552-15485030629376 type 0
Found missing chunk 15485030629376-15486104371200 type 0
Found missing chunk 15486104371200-15487178113024 type 0
Found missing chunk 15487178113024-15488251854848 type 0
Found missing chunk 15488251854848-15489325596672 type 0
Found missing chunk 15671861706752-15672935448576 type 0
Found missing chunk 15672935448576-15674009190400 type 0
Found missing chunk 15772793438208-15773867180032 type 0
Found missing chunk 15773867180032-15774940921856 type 0
Found missing chunk 15774940921856-15776014663680 type 0
Found missing chunk 15776014663680-15777088405504 type 0
Found missing chunk 15777088405504-15778162147328 type 0

Program received signal SIGSEGV, Segmentation fault.
find_free_extent (profile=2, exclude_nr=0, exclude_start=0, ins=0x7fffffffd760, hint_byte=<optimized out>, 
    search_end=18446744073709551615, search_start=0, empty_size=0, num_bytes=16384, orig_root=0x0, trans=0x5555556620f0)
    at kernel-shared/extent-tree.c:2249
2249            ret = find_search_start(root, &block_group, &search_start,
(gdb) bt
#0  find_free_extent (profile=2, exclude_nr=0, exclude_start=0, ins=0x7fffffffd760, hint_byte=<optimized out>, 
    search_end=18446744073709551615, search_start=0, empty_size=0, num_bytes=16384, orig_root=0x0, trans=0x5555556620f0)
    at kernel-shared/extent-tree.c:2249
#1  btrfs_reserve_extent (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, num_bytes=num_bytes@entry=16384, 
    empty_size=empty_size@entry=0, hint_byte=hint_byte@entry=0, search_end=search_end@entry=18446744073709551615, 
    ins=0x7fffffffd760, is_data=false) at kernel-shared/extent-tree.c:2380
#2  0x0000555555582eba in alloc_tree_block (ins=0x7fffffffd760, search_end=18446744073709551615, hint_byte=0, empty_size=0, level=0, 
    key=0x7fffffffd880, flags=0, generation=<optimized out>, root_objectid=3, num_bytes=16384, root=0x555555650030, 
    trans=0x5555556620f0) at kernel-shared/extent-tree.c:2515
#3  btrfs_alloc_free_block (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, blocksize=16384, root_objectid=3, 
    key=key@entry=0x7fffffffd880, level=level@entry=0, hint=0, empty_size=0) at kernel-shared/extent-tree.c:2570
#4  0x000055555557243f in __btrfs_cow_block (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, 
    buf=buf@entry=0x555555652b90, parent=parent@entry=0x0, parent_slot=parent_slot@entry=0, cow_ret=cow_ret@entry=0x7fffffffdab8, 
    search_start=0, empty_size=0) at kernel-shared/ctree.c:451
#5  0x0000555555572d2e in btrfs_cow_block (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, buf=0x555555652b90, 
    parent=0x0, parent_slot=0, cow_ret=cow_ret@entry=0x7fffffffdab8) at kernel-shared/ctree.c:544
#6  0x0000555555575730 in btrfs_search_slot (trans=0x5555556620f0, root=root@entry=0x555555650030, key=key@entry=0x55555565e030, 
    p=p@entry=0x555555662180, ins_len=ins_len@entry=105, cow=cow@entry=1) at kernel-shared/ctree.c:1377
#7  0x0000555555576e85 in btrfs_insert_empty_items (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, 
    path=path@entry=0x555555662180, cpu_key=cpu_key@entry=0x55555565e030, data_size=data_size@entry=0x7fffffffdb9c, nr=nr@entry=1)
    at kernel-shared/ctree.c:2824
#8  0x0000555555577295 in btrfs_insert_empty_item (data_size=<optimized out>, key=0x55555565e030, path=0x555555662180, 
    root=0x555555650030, trans=0x5555556620f0) at ./kernel-shared/ctree.h:2780
#9  btrfs_insert_item (trans=trans@entry=0x5555556620f0, root=0x555555650030, cpu_key=cpu_key@entry=0x55555565e030, 
    data=0x55555565df50, data_size=80) at kernel-shared/ctree.c:2923
#10 0x00005555555e303f in restore_missing_chunks (fs_info=0x55555564fbc0) at ./kernel-shared/ctree.h:322
#11 btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-recover-chunks.c:456
#12 0x00005555555d7c2f in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#13 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645d40 <cmd_struct_rescue_recover_chunks>)
    at cmds/commands.h:125
#14 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#15 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#16 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01 22:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
@ 2022-06-01 23:04                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  2022-06-01 23:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-01 23:04 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 1, 2022 at 6:56 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Jun 01, 2022 at 06:54:31PM -0400, Josef Bacik wrote:
> > Ah right, I have to mock up free space since we can't read our normal
> > stuff.  Fixed, lets go again.  Thanks,
>
> Found missing chunk 15483956887552-15485030629376 type 0
> Found missing chunk 15485030629376-15486104371200 type 0
> Found missing chunk 15486104371200-15487178113024 type 0
> Found missing chunk 15487178113024-15488251854848 type 0
> Found missing chunk 15488251854848-15489325596672 type 0
> Found missing chunk 15671861706752-15672935448576 type 0
> Found missing chunk 15672935448576-15674009190400 type 0
> Found missing chunk 15772793438208-15773867180032 type 0
> Found missing chunk 15773867180032-15774940921856 type 0
> Found missing chunk 15774940921856-15776014663680 type 0
> Found missing chunk 15776014663680-15777088405504 type 0
> Found missing chunk 15777088405504-15778162147328 type 0
>

I swear there's so much tech-debt here.  Try again please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01 23:04                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
@ 2022-06-01 23:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
  2022-06-02  0:04                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-01 23:10 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 01, 2022 at 07:04:32PM -0400, Josef Bacik wrote:
> On Wed, Jun 1, 2022 at 6:56 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Wed, Jun 01, 2022 at 06:54:31PM -0400, Josef Bacik wrote:
> > > Ah right, I have to mock up free space since we can't read our normal
> > > stuff.  Fixed, lets go again.  Thanks,
> >
> > Found missing chunk 15483956887552-15485030629376 type 0
> > Found missing chunk 15485030629376-15486104371200 type 0
> > Found missing chunk 15486104371200-15487178113024 type 0
> > Found missing chunk 15487178113024-15488251854848 type 0
> > Found missing chunk 15488251854848-15489325596672 type 0
> > Found missing chunk 15671861706752-15672935448576 type 0
> > Found missing chunk 15672935448576-15674009190400 type 0
> > Found missing chunk 15772793438208-15773867180032 type 0
> > Found missing chunk 15773867180032-15774940921856 type 0
> > Found missing chunk 15774940921856-15776014663680 type 0
> > Found missing chunk 15776014663680-15777088405504 type 0
> > Found missing chunk 15777088405504-15778162147328 type 0
> >
> 
> I swear there's so much tech-debt here.  Try again please.  Thanks,

Well, hopefully we'll get to the bottom of it all, thanks again for all
your efforts on this.

Found missing chunk 15772793438208-15773867180032 type 0
Found missing chunk 15773867180032-15774940921856 type 0
Found missing chunk 15774940921856-15776014663680 type 0
Found missing chunk 15776014663680-15777088405504 type 0
Found missing chunk 15777088405504-15778162147328 type 0

Program received signal SIGSEGV, Segmentation fault.
find_free_extent (profile=2, exclude_nr=0, exclude_start=0, ins=0x7fffffffd760, hint_byte=<optimized out>, 
    search_end=18446744073709551615, search_start=0, empty_size=0, num_bytes=16384, root=0x555555650030, trans=0x5555556620f0)
    at kernel-shared/extent-tree.c:2248
2248            ret = find_search_start(root, &block_group, &search_start,
(gdb) bt
#0  find_free_extent (profile=2, exclude_nr=0, exclude_start=0, ins=0x7fffffffd760, hint_byte=<optimized out>, 
    search_end=18446744073709551615, search_start=0, empty_size=0, num_bytes=16384, root=0x555555650030, trans=0x5555556620f0)
    at kernel-shared/extent-tree.c:2248
#1  btrfs_reserve_extent (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, num_bytes=num_bytes@entry=16384, 
    empty_size=empty_size@entry=0, hint_byte=hint_byte@entry=0, search_end=search_end@entry=18446744073709551615, 
    ins=0x7fffffffd760, is_data=false) at kernel-shared/extent-tree.c:2379
#2  0x0000555555582eb0 in alloc_tree_block (ins=0x7fffffffd760, search_end=18446744073709551615, hint_byte=0, empty_size=0, level=0, 
    key=0x7fffffffd880, flags=0, generation=<optimized out>, root_objectid=3, num_bytes=16384, root=0x555555650030, 
    trans=0x5555556620f0) at kernel-shared/extent-tree.c:2514
#3  btrfs_alloc_free_block (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, blocksize=16384, root_objectid=3, 
    key=key@entry=0x7fffffffd880, level=level@entry=0, hint=0, empty_size=0) at kernel-shared/extent-tree.c:2569
#4  0x000055555557243f in __btrfs_cow_block (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, 
    buf=buf@entry=0x555555652b90, parent=parent@entry=0x0, parent_slot=parent_slot@entry=0, cow_ret=cow_ret@entry=0x7fffffffdab8, 
    search_start=0, empty_size=0) at kernel-shared/ctree.c:451
#5  0x0000555555572d2e in btrfs_cow_block (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, buf=0x555555652b90, 
    parent=0x0, parent_slot=0, cow_ret=cow_ret@entry=0x7fffffffdab8) at kernel-shared/ctree.c:544
#6  0x0000555555575730 in btrfs_search_slot (trans=0x5555556620f0, root=root@entry=0x555555650030, key=key@entry=0x55555565e030, 
    p=p@entry=0x555555662180, ins_len=ins_len@entry=105, cow=cow@entry=1) at kernel-shared/ctree.c:1377
#7  0x0000555555576e85 in btrfs_insert_empty_items (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, 
    path=path@entry=0x555555662180, cpu_key=cpu_key@entry=0x55555565e030, data_size=data_size@entry=0x7fffffffdb9c, nr=nr@entry=1)
    at kernel-shared/ctree.c:2824
#8  0x0000555555577295 in btrfs_insert_empty_item (data_size=<optimized out>, key=0x55555565e030, path=0x555555662180, 
    root=0x555555650030, trans=0x5555556620f0) at ./kernel-shared/ctree.h:2780
#9  btrfs_insert_item (trans=trans@entry=0x5555556620f0, root=0x555555650030, cpu_key=cpu_key@entry=0x55555565e030, 
    data=0x55555565df50, data_size=80) at kernel-shared/ctree.c:2923
#10 0x00005555555e3058 in restore_missing_chunks (fs_info=0x55555564fbc0) at ./kernel-shared/ctree.h:322
#11 btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-recover-chunks.c:457
#12 0x00005555555d7c25 in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#13 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645d40 <cmd_struct_rescue_recover_chunks>)
    at cmds/commands.h:125
#14 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#15 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#16 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-01 23:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
@ 2022-06-02  0:04                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  2022-06-02  0:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-02  0:04 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 1, 2022 at 7:10 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Jun 01, 2022 at 07:04:32PM -0400, Josef Bacik wrote:
> > On Wed, Jun 1, 2022 at 6:56 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Wed, Jun 01, 2022 at 06:54:31PM -0400, Josef Bacik wrote:
> > > > Ah right, I have to mock up free space since we can't read our normal
> > > > stuff.  Fixed, lets go again.  Thanks,
> > >
> > > Found missing chunk 15483956887552-15485030629376 type 0
> > > Found missing chunk 15485030629376-15486104371200 type 0
> > > Found missing chunk 15486104371200-15487178113024 type 0
> > > Found missing chunk 15487178113024-15488251854848 type 0
> > > Found missing chunk 15488251854848-15489325596672 type 0
> > > Found missing chunk 15671861706752-15672935448576 type 0
> > > Found missing chunk 15672935448576-15674009190400 type 0
> > > Found missing chunk 15772793438208-15773867180032 type 0
> > > Found missing chunk 15773867180032-15774940921856 type 0
> > > Found missing chunk 15774940921856-15776014663680 type 0
> > > Found missing chunk 15776014663680-15777088405504 type 0
> > > Found missing chunk 15777088405504-15778162147328 type 0
> > >
> >
> > I swear there's so much tech-debt here.  Try again please.  Thanks,
>
> Well, hopefully we'll get to the bottom of it all, thanks again for all
> your efforts on this.
>
> Found missing chunk 15772793438208-15773867180032 type 0
> Found missing chunk 15773867180032-15774940921856 type 0
> Found missing chunk 15774940921856-15776014663680 type 0
> Found missing chunk 15776014663680-15777088405504 type 0
> Found missing chunk 15777088405504-15778162147328 type 0
>

This segfault makes no sense, we check to make sure any of this stuff
is NULL.  I've added some debugging, hopefully that'll shed some
light.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02  0:04                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
@ 2022-06-02  0:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  2022-06-02  1:23                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-02  0:06 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 01, 2022 at 08:04:03PM -0400, Josef Bacik wrote:
> > Found missing chunk 15772793438208-15773867180032 type 0
> > Found missing chunk 15773867180032-15774940921856 type 0
> > Found missing chunk 15774940921856-15776014663680 type 0
> > Found missing chunk 15776014663680-15777088405504 type 0
> > Found missing chunk 15777088405504-15778162147328 type 0
> 
> This segfault makes no sense, we check to make sure any of this stuff
> is NULL.  I've added some debugging, hopefully that'll shed some
> light.  Thanks,

Found missing chunk 15486104371200-15487178113024 type 0
Found missing chunk 15487178113024-15488251854848 type 0
Found missing chunk 15488251854848-15489325596672 type 0
Found missing chunk 15671861706752-15672935448576 type 0
Found missing chunk 15672935448576-15674009190400 type 0
Found missing chunk 15772793438208-15773867180032 type 0
Found missing chunk 15773867180032-15774940921856 type 0
Found missing chunk 15774940921856-15776014663680 type 0
Found missing chunk 15776014663680-15777088405504 type 0
Found missing chunk 15777088405504-15778162147328 type 0
adding bg for 20971520 8388608

Program received signal SIGSEGV, Segmentation fault.
0x00005555555826d5 in cache_block_group (block_group=0x555555662000, root=0x0) at ./kerncompat.h:133
133             abort();
(gdb) bt
#0  0x00005555555826d5 in cache_block_group (block_group=0x555555662000, root=0x0) at ./kerncompat.h:133
#1  find_search_start (profile=2, num=16384, start_ret=<synthetic pointer>, cache_ret=<synthetic pointer>, root=0x555555650030)
    at kernel-shared/extent-tree.c:280
#2  find_free_extent (profile=2, exclude_nr=0, exclude_start=0, ins=0x7fffffffd760, hint_byte=<optimized out>, 
    search_end=18446744073709551615, search_start=0, empty_size=0, num_bytes=16384, root=0x555555650030, trans=0x5555556620e0)
    at kernel-shared/extent-tree.c:2254
#3  btrfs_reserve_extent (trans=trans@entry=0x5555556620e0, root=root@entry=0x555555650030, num_bytes=num_bytes@entry=16384, 
    empty_size=empty_size@entry=0, hint_byte=hint_byte@entry=0, search_end=search_end@entry=18446744073709551615, 
    ins=0x7fffffffd760, is_data=false) at kernel-shared/extent-tree.c:2385
#4  0x0000555555582f12 in alloc_tree_block (ins=0x7fffffffd760, search_end=18446744073709551615, hint_byte=0, empty_size=0, level=0, 
    key=0x7fffffffd880, flags=0, generation=<optimized out>, root_objectid=3, num_bytes=16384, root=0x555555650030, 
    trans=0x5555556620e0) at kernel-shared/extent-tree.c:2520
#5  btrfs_alloc_free_block (trans=trans@entry=0x5555556620e0, root=root@entry=0x555555650030, blocksize=16384, root_objectid=3, 
    key=key@entry=0x7fffffffd880, level=level@entry=0, hint=0, empty_size=0) at kernel-shared/extent-tree.c:2575
#6  0x000055555557243f in __btrfs_cow_block (trans=trans@entry=0x5555556620e0, root=root@entry=0x555555650030, 
    buf=buf@entry=0x555555652b90, parent=parent@entry=0x0, parent_slot=parent_slot@entry=0, cow_ret=cow_ret@entry=0x7fffffffdab8, 
    search_start=0, empty_size=0) at kernel-shared/ctree.c:451
#7  0x0000555555572d2e in btrfs_cow_block (trans=trans@entry=0x5555556620e0, root=root@entry=0x555555650030, buf=0x555555652b90, 
    parent=0x0, parent_slot=0, cow_ret=cow_ret@entry=0x7fffffffdab8) at kernel-shared/ctree.c:544
#8  0x0000555555575730 in btrfs_search_slot (trans=0x5555556620e0, root=root@entry=0x555555650030, key=key@entry=0x55555564fb70, 
    p=p@entry=0x555555662170, ins_len=ins_len@entry=105, cow=cow@entry=1) at kernel-shared/ctree.c:1377
#9  0x0000555555576e85 in btrfs_insert_empty_items (trans=trans@entry=0x5555556620e0, root=root@entry=0x555555650030, 
    path=path@entry=0x555555662170, cpu_key=cpu_key@entry=0x55555564fb70, data_size=data_size@entry=0x7fffffffdb9c, nr=nr@entry=1)
    at kernel-shared/ctree.c:2824
#10 0x0000555555577295 in btrfs_insert_empty_item (data_size=<optimized out>, key=0x55555564fb70, path=0x555555662170, 
    root=0x555555650030, trans=0x5555556620e0) at ./kernel-shared/ctree.h:2780
#11 btrfs_insert_item (trans=trans@entry=0x5555556620e0, root=0x555555650030, cpu_key=cpu_key@entry=0x55555564fb70, 
    data=0x55555565de80, data_size=80) at kernel-shared/ctree.c:2923
#12 0x00005555555e30dc in restore_missing_chunks (fs_info=0x55555564fbc0) at ./kernel-shared/ctree.h:322
#13 btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-recover-chunks.c:458
#14 0x00005555555d7c87 in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#15 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645d40 <cmd_struct_rescue_recover_chunks>)
    at cmds/commands.h:125
#16 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#17 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#18 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02  0:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
@ 2022-06-02  1:23                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  2022-06-02  1:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-02  1:23 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 1, 2022 at 8:06 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Jun 01, 2022 at 08:04:03PM -0400, Josef Bacik wrote:
> > > Found missing chunk 15772793438208-15773867180032 type 0
> > > Found missing chunk 15773867180032-15774940921856 type 0
> > > Found missing chunk 15774940921856-15776014663680 type 0
> > > Found missing chunk 15776014663680-15777088405504 type 0
> > > Found missing chunk 15777088405504-15778162147328 type 0
> >
> > This segfault makes no sense, we check to make sure any of this stuff
> > is NULL.  I've added some debugging, hopefully that'll shed some
> > light.  Thanks,
>
> Found missing chunk 15486104371200-15487178113024 type 0
> Found missing chunk 15487178113024-15488251854848 type 0
> Found missing chunk 15488251854848-15489325596672 type 0
> Found missing chunk 15671861706752-15672935448576 type 0
> Found missing chunk 15672935448576-15674009190400 type 0
> Found missing chunk 15772793438208-15773867180032 type 0
> Found missing chunk 15773867180032-15774940921856 type 0
> Found missing chunk 15774940921856-15776014663680 type 0
> Found missing chunk 15776014663680-15777088405504 type 0
> Found missing chunk 15777088405504-15778162147328 type 0
> adding bg for 20971520 8388608
>

Ah ok I'm not confused anymore, try that now.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02  1:23                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
@ 2022-06-02  1:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  2022-06-02  2:03                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-02  1:55 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 01, 2022 at 09:23:30PM -0400, Josef Bacik wrote:
> Ah ok I'm not confused anymore, try that now.  Thanks,

Better:

(...)
Found missing chunk 15672935448576-15674009190400 type 0
Found missing chunk 15772793438208-15773867180032 type 0
Found missing chunk 15773867180032-15774940921856 type 0
Found missing chunk 15774940921856-15776014663680 type 0
Found missing chunk 15776014663680-15777088405504 type 0
Found missing chunk 15777088405504-15778162147328 type 0
adding bg for 20971520 8388608
Unable to find block group for 0
Unable to find block group for 0
Unable to find block group for 0
Inserting chunk 14823605665792wtf transid 2582704 ret -28
ERROR: Couldn't insert item
doing close???
ERROR: attempt to start transaction over already running one
extent buffer leak: start 22593536 len 16384
Recover chunks tree failed
[Inferior 1 (process 1930) exited with code 0344]

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02  1:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
@ 2022-06-02  2:03                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  2022-06-02  2:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-02  2:03 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 1, 2022 at 9:55 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Jun 01, 2022 at 09:23:30PM -0400, Josef Bacik wrote:
> > Ah ok I'm not confused anymore, try that now.  Thanks,
>
> Better:
>

Ah right, we don't actually use the normal free space cache.  Fixed
that up, thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02  2:03                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
@ 2022-06-02  2:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  2022-06-02 14:07                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-02  2:16 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 01, 2022 at 10:03:29PM -0400, Josef Bacik wrote:
> On Wed, Jun 1, 2022 at 9:55 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Wed, Jun 01, 2022 at 09:23:30PM -0400, Josef Bacik wrote:
> > > Ah ok I'm not confused anymore, try that now.  Thanks,
> >
> > Better:
> >
> 
> Ah right, we don't actually use the normal free space cache.  Fixed
> that up, thanks,

ound missing chunk 14400551387136-14401625128960 type 0
Found missing chunk 14401625128960-14402698870784 type 0
Found missing chunk 14402698870784-14403772612608 type 0
Found missing chunk 14403772612608-14404846354432 type 0
Found missing chunk 14404846354432-14405920096256 type 0
Found missing chunk 14405920096256-14406993838080 type 0
ing chunk 11143322009600Inserting chunk 11144395751424Inserting chunk 11145469493248Inserting chunk 11146543235072Inserting chunk 11147616976896Inserting chunk 11148690718720Inserting chunk 11149764460544Inserting chunk 11150838202368Inserting chunk 11151911944192Inserting chunk 11152985686016Inserting chunk 11154059427840Inserting chunk 11155133169664Inserting chunk 11156206911488Inserting chunk 11157280653312Inserting chunk 11159428136960Inserting chunk 11160501878784Inserting chunk 11161038749696Inserting chunk 11162112491520Inserting chunk 11163186233344Inserting chunk (...)
g chunk 15323969355776Inserting chunk 15325043097600Inserting chunk 15326116839424Inserting chunk 15327190581248Inserting chunk 15328264323072Inserting chunk 15329338064896Inserting chunk 15332559290368Inserting chunk 15333633032192Inserting chunk 15334706774016Inserting chunk 15355107868672Inserting chunk 15356181610496Inserting chunk 15357255352320Inserting chunk 15358329094144Inserting chunk 15359402835968Inserting chunk 15360476577792Inserting chunk 15361550319616Inserting chunk 15362624061440Inserting chunk 15363697803264Inserting chunk 15364771545088Inserting chunk 15365845286912Inserting chunk 15366919028736Inserting chunk 15395910057984Inserting chunk 15396983799808Inserting chunk 15400205025280Inserting chunk 15401278767104Inserting chunk 15402352508928Inserting chunk 15405573734400Inserting chunk 15408794959872Inserting chunk 15409868701696Inserting chunk 15410942443520Inserting chunk 15412016185344Inserting chunk 15413089927168Inserting chunk 15414163668992Inserting chunk 15415237410816Inserting chunk 15416311152640Inserting chunk 15417384894464Inserting chunk 15418458636288Inserting chunk 15419532378112Inserting chunk 15420606119936Inserting chunk 15421679861760Inserting chunk 15422753603584Inserting chunk 15423827345408Inserting chunk 15424901087232Inserting chunk 15425974829056Inserting chunk 15427048570880Inserting chunk 15428122312704Inserting chunk 15429196054528Inserting chunk 15430269796352Inserting chunk 15431343538176Inserting chunk 15432417280000Inserting chunk 15433491021824Inserting chunk 15434564763648Inserting chunk 15435638505472Inserting chunk 15436712247296Inserting chunk 15437785989120Inserting chunk 15438859730944Inserting chunk 15439933472768Inserting chunk 15441007214592Inserting chunk 15442080956416Inserting chunk 15443154698240Inserting chunk 15444228440064Inserting chunk 15445302181888Inserting chunk 15446375923712Inserting chunk 15447449665536Inserting chunk 15448523407360Inserting chunk 15449597149184Inserting chunk 15450670891008Inserting chunk 15451744632832Inserting chunk 15452818374656Inserting chunk 15453892116480Inserting chunk 15454965858304Inserting chunk 15456039600128Inserting chunk 15457113341952Inserting chunk 15458187083776Inserting chunk 15459260825600Inserting chunk 15460334567424Inserting chunk 15461408309248Inserting chunk 15462482051072Inserting chunk 15463555792896Inserting chunk 15464629534720Inserting chunk 15465703276544Inserting chunk 15466777018368Inserting chunk 15467850760192Inserting chunk 15468924502016Inserting chunk 15469998243840Inserting chunk 15471071985664Inserting chunk 15472145727488Inserting chunk 15473219469312Inserting chunk 15474293211136Inserting chunk 15475366952960Inserting chkernel-shared/extent-tree.c:2417: alloc_reserved_tree_block: Assertion `sinfo` failed, value 0
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x2981a)[0x55555557d81a]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_run_delayed_refs+0x531)[0x555555583998]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_commit_transaction+0x3b)[0x5555555928a8]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_find_recover_chunks+0x594)[0x5555555e311d]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x83c87)[0x5555555d7c87]
/var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b]
/var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd]
/var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]

Program received signal SIGABRT, Aborted.
0x00007ffff78768a1 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff78768a1 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff7860546 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x000055555558399d in assert_trace (val=0, line=2417, func=<synthetic pointer>, 
    filename=0x55555560cef2 "kernel-shared/extent-tree.c", assertion=0x55555560d04b "sinfo") at ./kerncompat.h:338
#3  alloc_reserved_tree_block (extent_op=0x555555652ae0, node=0x555555651ab0, trans=0x5555556620e0)
    at kernel-shared/extent-tree.c:2417
#4  run_delayed_tree_ref (insert_reserved=<optimized out>, extent_op=0x555555652ae0, node=0x555555651ab0, fs_info=0x55555564fbc0, 
    trans=0x5555556620e0) at kernel-shared/extent-tree.c:3738
#5  run_one_delayed_ref (insert_reserved=<optimized out>, extent_op=0x555555652ae0, node=0x555555651ab0, fs_info=0x55555564fbc0, 
    trans=0x5555556620e0) at kernel-shared/extent-tree.c:3761
#6  btrfs_run_delayed_refs (trans=trans@entry=0x5555556620e0, nr=nr@entry=18446744073709551615) at kernel-shared/extent-tree.c:3845
#7  0x00005555555928a8 in btrfs_commit_transaction (trans=trans@entry=0x5555556620e0, root=0x555555650030)
    at kernel-shared/transaction.c:181
#8  0x00005555555e311d in restore_missing_chunks (fs_info=0x55555564fbc0) at cmds/rescue-recover-chunks.c:405
#9  btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-recover-chunks.c:460
#10 0x00005555555d7c87 in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#11 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645d40 <cmd_struct_rescue_recover_chunks>)
    at cmds/commands.h:125
#12 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#13 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#14 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
(gdb) 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02  2:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
@ 2022-06-02 14:07                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  2022-06-02 14:21                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-02 14:07 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 1, 2022 at 10:16 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Jun 01, 2022 at 10:03:29PM -0400, Josef Bacik wrote:
> > On Wed, Jun 1, 2022 at 9:55 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Wed, Jun 01, 2022 at 09:23:30PM -0400, Josef Bacik wrote:
> > > > Ah ok I'm not confused anymore, try that now.  Thanks,
> > >
> > > Better:
> > >
> >
> > Ah right, we don't actually use the normal free space cache.  Fixed
> > that up, thanks,
>
> ound missing chunk 14400551387136-14401625128960 type 0
> Found missing chunk 14401625128960-14402698870784 type 0
> Found missing chunk 14402698870784-14403772612608 type 0
> Found missing chunk 14403772612608-14404846354432 type 0
> Found missing chunk 14404846354432-14405920096256 type 0
> Found missing chunk 14405920096256-14406993838080 type 0
> ing chunk 11143322009600Inserting chunk 11144395751424Inserting chunk 11145469493248Inserting chunk 11146543235072Inserting chunk 11147616976896Inserting chunk 11148690718720Inserting chunk 11149764460544Inserting chunk 11150838202368Inserting chunk 11151911944192Inserting chunk 11152985686016Inserting chunk 11154059427840Inserting chunk 11155133169664Inserting chunk 11156206911488Inserting chunk 11157280653312Inserting chunk 11159428136960Inserting chunk 11160501878784Inserting chunk 11161038749696Inserting chunk 11162112491520Inserting chunk 11163186233344Inserting chunk (...)
> g chunk 15323969355776Inserting chunk 15325043097600Inserting chunk 15326116839424Inserting chunk 15327190581248Inserting chunk 15328264323072Inserting chunk 15329338064896Inserting chunk 15332559290368Inserting chunk 15333633032192Inserting chunk 15334706774016Inserting chunk 15355107868672Inserting chunk 15356181610496Inserting chunk 15357255352320Inserting chunk 15358329094144Inserting chunk 15359402835968Inserting chunk 15360476577792Inserting chunk 15361550319616Inserting chunk 15362624061440Inserting chunk 15363697803264Inserting chunk 15364771545088Inserting chunk 15365845286912Inserting chunk 15366919028736Inserting chunk 15395910057984Inserting chunk 15396983799808Inserting chunk 15400205025280Inserting chunk 15401278767104Inserting chunk 15402352508928Inserting chunk 15405573734400Inserting chunk 15408794959872Inserting chunk 15409868701696Inserting chunk 15410942443520Inserting chunk 15412016185344Inserting chunk 15413089927168Inserting chunk 15414163668992Inserting chunk 15415237410816Inserting chunk 15416311152640Inserting chunk 15417384894464Inserting chunk 15418458636288Inserting chunk 15419532378112Inserting chunk 15420606119936Inserting chunk 15421679861760Inserting chunk 15422753603584Inserting chunk 15423827345408Inserting chunk 15424901087232Inserting chunk 15425974829056Inserting chunk 15427048570880Inserting chunk 15428122312704Inserting chunk 15429196054528Inserting chunk 15430269796352Inserting chunk 15431343538176Inserting chunk 15432417280000Inserting chunk 15433491021824Inserting chunk 15434564763648Inserting chunk 15435638505472Inserting chunk 15436712247296Inserting chunk 15437785989120Inserting chunk 15438859730944Inserting chunk 15439933472768Inserting chunk 15441007214592Inserting chunk 15442080956416Inserting chunk 15443154698240Inserting chunk 15444228440064Inserting chunk 15445302181888Inserting chunk 15446375923712Inserting chunk 15447449665536Inserting chunk 15448523407360Inserting chunk 15449597149184Inserting chunk 15450670891008Inserting chunk 15451744632832Inserting chunk 15452818374656Inserting chunk 15453892116480Inserting chunk 15454965858304Inserting chunk 15456039600128Inserting chunk 15457113341952Inserting chunk 15458187083776Inserting chunk 15459260825600Inserting chunk 15460334567424Inserting chunk 15461408309248Inserting chunk 15462482051072Inserting chunk 15463555792896Inserting chunk 15464629534720Inserting chunk 15465703276544Inserting chunk 15466777018368Inserting chunk 15467850760192Inserting chunk 15468924502016Inserting chunk 15469998243840Inserting chunk 15471071985664Inserting chunk 15472145727488Inserting chunk 15473219469312Inserting chunk 15474293211136Inserting chunk 15475366952960Inserting

Woo we're in the transaction commit now boys, try again please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02 14:07                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
@ 2022-06-02 14:21                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
  2022-06-02 14:27                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-02 14:21 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Jun 02, 2022 at 10:07:27AM -0400, Josef Bacik wrote:
> > g chunk 15323969355776Inserting chunk 15325043097600Inserting chunk 15326116839424Inserting chunk 15327190581248Inserting chunk 15328264323072Inserting chunk 15329338064896Inserting chunk 15332559290368Inserting chunk 15333633032192Inserting chunk 15334706774016Inserting chunk 15355107868672Inserting chunk 15356181610496Inserting chunk 15357255352320Inserting chunk 15358329094144Inserting chunk 15359402835968Inserting chunk 15360476577792Inserting chunk 15361550319616Inserting chunk 15362624061440Inserting chunk 15363697803264Inserting chunk 15364771545088Inserting chunk 15365845286912Inserting chunk 15366919028736Inserting chunk 15395910057984Inserting chunk 15396983799808Inserting chunk 15400205025280Inserting chunk 15401278767104Inserting chunk 15402352508928Inserting chunk 15405573734400Inserting chunk 15408794959872Inserting chunk 15409868701696Inserting chunk 15410942443520Inserting chunk 15412016185344Inserting chunk 15413089927168Inserting chunk 15414163668992Inserting chunk 15415237410816Inserting chunk 15416311152640Inserting chunk 15417384894464Inserting chunk 15418458636288Inserting chunk 15419532378112Inserting chunk 15420606119936Inserting chunk 15421679861760Inserting chunk 15422753603584Inserting chunk 15423827345408Inserting chunk 15424901087232Inserting chunk 15425974829056Inserting chunk 15427048570880Inserting chunk 15428122312704Inserting chunk 15429196054528Inserting chunk 15430269796352Inserting chunk 15431343538176Inserting chunk 15432417280000Inserting chunk 15433491021824Inserting chunk 15434564763648Inserting chunk 15435638505472Inserting chunk 15436712247296Inserting chunk 15437785989120Inserting chunk 15438859730944Inserting chunk 15439933472768Inserting chunk 15441007214592Inserting chunk 15442080956416Inserting chunk 15443154698240Inserting chunk 15444228440064Inserting chunk 15445302181888Inserting chunk 15446375923712Inserting chunk 15447449665536Inserting chunk 15448523407360Inserting chunk 15449597149184Inserting chunk 15450670891008Inserting chunk 15451744632832Inserting chunk 15452818374656Inserting chunk 15453892116480Inserting chunk 15454965858304Inserting chunk 15456039600128Inserting chunk 15457113341952Inserting chunk 15458187083776Inserting chunk 15459260825600Inserting chunk 15460334567424Inserting chunk 15461408309248Inserting chunk 15462482051072Inserting chunk 15463555792896Inserting chunk 15464629534720Inserting chunk 15465703276544Inserting chunk 15466777018368Inserting chunk 15467850760192Inserting chunk 15468924502016Inserting chunk 15469998243840Inserting chunk 15471071985664Inserting chunk 15472145727488Inserting chunk 15473219469312Inserting chunk 15474293211136Inserting chunk 15475366952960Inserting
> 
> Woo we're in the transaction commit now boys, try again please.  Thanks,

Sure.

Note that you're missing some newlines in the output:

7296Inserting chunk 15437785989120Inserting chunk 15438859730944Inserting chunk 15439933472768Inserting chunk 15441007214592Inserting chunk 15442080956416Inserting chunk 15443154698240Inserting chunk 15444228440064Inserting chunk 15445302181888Inserting chunk 15446375923712Inserting chunk 15447449665536Inserting chunk 15448523407360Inserting chunk 15449597149184Inserting chunk 15450670891008Inserting chunk 15451744632832Inserting chunk 15452818374656Inserting chunk 15453892116480Inserting chunk 15454965858304Inserting chunk 15456039600128Inserting chunk 15457113341952Inserting chunk 15458187083776Inserting chunk 15459260825600Inserting chunk 15460334567424Inserting chunk 15461408309248Inserting chunk 15462482051072Inserting chunk 15463555792896Inserting chunk 15464629534720Inserting chunk 15465703276544Inserting chunk 15466777018368Inserting chunk 15467850760192Inserting chunk 15468924502016Inserting chunk 15469998243840Inserting chunk 15471071985664Inserting chunk 15472145727488Inserting chunk 15473219469312Inserting chunk 15474293211136Inserting chunk 15475366952960Inserting chkernel-shared/extent-tree.c:2417: alloc_reserved_tree_block: Assertion `sinfo` failed, value 0
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x2981a)[0x55555557d81a]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_run_delayed_refs+0x531)[0x555555583998]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_commit_transaction+0x3b)[0x5555555928a8]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_find_recover_chunks+0x594)[0x5555555e311d]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x83c87)[0x5555555d7c87]
/var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b]
/var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd]
/var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]

Program received signal SIGABRT, Aborted.
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02 14:21                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
@ 2022-06-02 14:27                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  2022-06-02 14:36                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-02 14:27 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, Jun 2, 2022 at 10:21 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Jun 02, 2022 at 10:07:27AM -0400, Josef Bacik wrote:
> > > g chunk 15323969355776Inserting chunk 15325043097600Inserting chunk 15326116839424Inserting chunk 15327190581248Inserting chunk 15328264323072Inserting chunk 15329338064896Inserting chunk 15332559290368Inserting chunk 15333633032192Inserting chunk 15334706774016Inserting chunk 15355107868672Inserting chunk 15356181610496Inserting chunk 15357255352320Inserting chunk 15358329094144Inserting chunk 15359402835968Inserting chunk 15360476577792Inserting chunk 15361550319616Inserting chunk 15362624061440Inserting chunk 15363697803264Inserting chunk 15364771545088Inserting chunk 15365845286912Inserting chunk 15366919028736Inserting chunk 15395910057984Inserting chunk 15396983799808Inserting chunk 15400205025280Inserting chunk 15401278767104Inserting chunk 15402352508928Inserting chunk 15405573734400Inserting chunk 15408794959872Inserting chunk 15409868701696Inserting chunk 15410942443520Inserting chunk 15412016185344Inserting chunk 15413089927168Inserting chunk 15414163668992Inserting chunk 15415237410816Inserting chunk 15416311152640Inserting chunk 15417384894464Inserting chunk 15418458636288Inserting chunk 15419532378112Inserting chunk 15420606119936Inserting chunk 15421679861760Inserting chunk 15422753603584Inserting chunk 15423827345408Inserting chunk 15424901087232Inserting chunk 15425974829056Inserting chunk 15427048570880Inserting chunk 15428122312704Inserting chunk 15429196054528Inserting chunk 15430269796352Inserting chunk 15431343538176Inserting chunk 15432417280000Inserting chunk 15433491021824Inserting chunk 15434564763648Inserting chunk 15435638505472Inserting chunk 15436712247296Inserting chunk 15437785989120Inserting chunk 15438859730944Inserting chunk 15439933472768Inserting chunk 15441007214592Inserting chunk 15442080956416Inserting chunk 15443154698240Inserting chunk 15444228440064Inserting chunk 15445302181888Inserting chunk 15446375923712Inserting chunk 15447449665536Inserting chunk 15448523407360Inserting chunk 15449597149184Inserting chunk 15450670891008Inserting chunk 15451744632832Inserting chunk 15452818374656Inserting chunk 15453892116480Inserting chunk 15454965858304Inserting chunk 15456039600128Inserting chunk 15457113341952Inserting chunk 15458187083776Inserting chunk 15459260825600Inserting chunk 15460334567424Inserting chunk 15461408309248Inserting chunk 15462482051072Inserting chunk 15463555792896Inserting chunk 15464629534720Inserting chunk 15465703276544Inserting chunk 15466777018368Inserting chunk 15467850760192Inserting chunk 15468924502016Inserting chunk 15469998243840Inserting chunk 15471071985664Inserting chunk 15472145727488Inserting chunk 15473219469312Inserting chunk 15474293211136Inserting chunk 15475366952960Inserting
> >
> > Woo we're in the transaction commit now boys, try again please.  Thanks,
>
> Sure.
>
> Note that you're missing some newlines in the output:
>

Ooops, helps if I use the new flag I added.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02 14:27                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
@ 2022-06-02 14:36                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  2022-06-02 18:43                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-02 14:36 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Jun 02, 2022 at 10:27:59AM -0400, Josef Bacik wrote:
> 
> Ooops, helps if I use the new flag I added.  Thanks,

Good news?

Inserting chunk 15486104371200
Inserting chunk 15487178113024
Inserting chunk 15488251854848
Inserting chunk 15671861706752
Inserting chunk 15672935448576
Inserting chunk 15772793438208
Inserting chunk 15773867180032
Inserting chunk 15774940921856
Inserting chunk 15776014663680
Inserting chunk 15777088405504
WARNING: reserved space leaked, transid=2582704 flag=0x2
bytes_reserved=180224
doing close???
WARNING: reserved space leaked, flag=0x2 bytes_reserved=180224
Recover chunks succeeded, you can run check now
[Inferior 1 (process 21907) exited normally]

What check do I run now?
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02 14:36                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
@ 2022-06-02 18:43                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  2022-06-02 19:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-02 18:43 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, Jun 2, 2022 at 10:36 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Jun 02, 2022 at 10:27:59AM -0400, Josef Bacik wrote:
> >
> > Ooops, helps if I use the new flag I added.  Thanks,
>
> Good news?
>
> Inserting chunk 15486104371200
> Inserting chunk 15487178113024
> Inserting chunk 15488251854848
> Inserting chunk 15671861706752
> Inserting chunk 15672935448576
> Inserting chunk 15772793438208
> Inserting chunk 15773867180032
> Inserting chunk 15774940921856
> Inserting chunk 15776014663680
> Inserting chunk 15777088405504
> WARNING: reserved space leaked, transid=2582704 flag=0x2
> bytes_reserved=180224
> doing close???
> WARNING: reserved space leaked, flag=0x2 bytes_reserved=180224
> Recover chunks succeeded, you can run check now
> [Inferior 1 (process 21907) exited normally]
>
> What check do I run now?

Now we run

btrfs rescue tree-recover <device>

to restore the roots we lost.  After that we run

btrfs rescue init-extent-tree <device>

then

btrfs rescue init-csum-tree <device>

and then finally we run

btrfs check <device>

Now I have to write some code to fix up the device extents for the
chunks I just added back, but I need to make sure that's the only
thing check complains about.  Once we have that worked out I'll write
the code to add the device extents for the restored chunks and then
theoretically we'll be done.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02 18:43                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
@ 2022-06-02 19:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  2022-06-02 19:35                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-02 19:08 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Jun 02, 2022 at 02:43:03PM -0400, Josef Bacik wrote:
> Now we run
> 
> btrfs rescue tree-recover <device>
 
It got pretty far, and then
Invalid mapping for 364837339136-364837355520, got 11106814787584-11107888529408
Couldn't map the block 364837339136
Couldn't map the block 364837339136
deleting slot 0 in block 11160501878784
Invalid mapping for 364837306368-364837322752, got 11106814787584-11107888529408
Couldn't map the block 364837306368
Couldn't map the block 364837306368
deleting slot 0 in block 11160501878784
Invalid mapping for 364746457088-364746473472, got 11106814787584-11107888529408
Couldn't map the block 364746457088
Couldn't map the block 364746457088
deleting slot 0 in block 11160501878784
bad tree block 11160501878784, invalid nr_items: 0
kernel-shared/disk-io.c:553: write_tree_block: BUG_ON `1` triggered, value 1
./btrfs(+0x25ae7)[0x55c9d8c1bae7]
./btrfs(write_tree_block+0xb8)[0x55c9d8c1dec9]
./btrfs(+0x8aa38)[0x55c9d8c80a38]
./btrfs(+0x8acbe)[0x55c9d8c80cbe]
./btrfs(+0x8adcb)[0x55c9d8c80dcb]
./btrfs(btrfs_recover_trees+0x628)[0x55c9d8c81a3b]
./btrfs(+0x83f1f)[0x55c9d8c79f1f]
./btrfs(handle_command_group+0x49)[0x55c9d8c0e17b]
./btrfs(main+0x94)[0x55c9d8c0e275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fe2988107fd]
./btrfs(_start+0x2a)[0x55c9d8c0de1a]
Aborted


> to restore the roots we lost.  After that we run
> 
> btrfs rescue init-extent-tree <device>
> 
> then
> 
> btrfs rescue init-csum-tree <device>
> 
> and then finally we run
> 
> btrfs check <device>
> 
> Now I have to write some code to fix up the device extents for the
> chunks I just added back, but I need to make sure that's the only
> thing check complains about.  Once we have that worked out I'll write
> the code to add the device extents for the restored chunks and then
> theoretically we'll be done.  Thanks,
> 
> Josef
> 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02 19:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
@ 2022-06-02 19:35                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  2022-06-02 19:51                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-02 19:35 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, Jun 2, 2022 at 3:08 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Jun 02, 2022 at 02:43:03PM -0400, Josef Bacik wrote:
> > Now we run
> >
> > btrfs rescue tree-recover <device>
>
> It got pretty far, and then
> Invalid mapping for 364837339136-364837355520, got 11106814787584-11107888529408
> Couldn't map the block 364837339136
> Couldn't map the block 364837339136
> deleting slot 0 in block 11160501878784
> Invalid mapping for 364837306368-364837322752, got 11106814787584-11107888529408
> Couldn't map the block 364837306368
> Couldn't map the block 364837306368
> deleting slot 0 in block 11160501878784
> Invalid mapping for 364746457088-364746473472, got 11106814787584-11107888529408
> Couldn't map the block 364746457088
> Couldn't map the block 364746457088
> deleting slot 0 in block 11160501878784

Was it printing a lot of these messages?  I was sort of hoping we
found all the chunks so it didn't feel the need to delete a bunch of
stuff.  Can you re-run

btrfs rescue recover-chunks <device>

and make sure it doesn't find anything new?  Maybe there were some
system chunks that it found that has the other chunks in it.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02 19:35                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
@ 2022-06-02 19:51                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  2022-06-02 19:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-02 19:51 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Jun 02, 2022 at 03:35:46PM -0400, Josef Bacik wrote:
> On Thu, Jun 2, 2022 at 3:08 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Thu, Jun 02, 2022 at 02:43:03PM -0400, Josef Bacik wrote:
> > > Now we run
> > >
> > > btrfs rescue tree-recover <device>
> >
> > It got pretty far, and then
> > Invalid mapping for 364837339136-364837355520, got 11106814787584-11107888529408
> > Couldn't map the block 364837339136
> > Couldn't map the block 364837339136
> > deleting slot 0 in block 11160501878784
> > Invalid mapping for 364837306368-364837322752, got 11106814787584-11107888529408
> > Couldn't map the block 364837306368
> > Couldn't map the block 364837306368
> > deleting slot 0 in block 11160501878784
> > Invalid mapping for 364746457088-364746473472, got 11106814787584-11107888529408
> > Couldn't map the block 364746457088
> > Couldn't map the block 364746457088
> > deleting slot 0 in block 11160501878784
> 
> Was it printing a lot of these messages?  I was sort of hoping we

Yes

> found all the chunks so it didn't feel the need to delete a bunch of
> stuff.  Can you re-run
> 
> btrfs rescue recover-chunks <device>
> 
> and make sure it doesn't find anything new?  Maybe there were some
> system chunks that it found that has the other chunks in it.  Thanks,

output is very long.  I was able to paste a good part in
https://justpaste.it/5cy5s

last lines:
trying bytenr 15645980491776 got 3 blocks 8 bad
Repairing root 163318 bad_blocks 8 update 1
Invalid mapping for 11822436614144-11822436630528, got 14271702368256-14272776110080
Couldn't map the block 11822436614144
Couldn't map the block 11822436614144
deleting slot 0 in block 15645980491776
Invalid mapping for 365011795968-365011812352, got 11106814787584-11107888529408
Couldn't map the block 365011795968
Couldn't map the block 365011795968
deleting slot 0 in block 15645980491776
Invalid mapping for 365043269632-365043286016, got 11106814787584-11107888529408
Couldn't map the block 365043269632
Couldn't map the block 365043269632
deleting slot 0 in block 15645980491776
Invalid mapping for 365031768064-365031784448, got 11106814787584-11107888529408
Couldn't map the block 365031768064
Couldn't map the block 365031768064
deleting slot 0 in block 15645980491776
Invalid mapping for 365028409344-365028425728, got 11106814787584-11107888529408
Couldn't map the block 365028409344
Couldn't map the block 365028409344
deleting slot 0 in block 15645980491776
Invalid mapping for 365033668608-365033684992, got 11106814787584-11107888529408
Couldn't map the block 365033668608
Couldn't map the block 365033668608
deleting slot 0 in block 15645980491776
deleting slot 0 in block 15645980491776
deleting slot 0 in block 15645980491776
bad tree block 15645980491776, invalid nr_items: 0
kernel-shared/disk-io.c:553: write_tree_block: BUG_ON `1` triggered, value 1
./btrfs(+0x25ae7)[0x558782bb4ae7]
./btrfs(write_tree_block+0xb8)[0x558782bb6ec9]
./btrfs(+0x8aa38)[0x558782c19a38]
./btrfs(+0x8adcb)[0x558782c19dcb]
./btrfs(btrfs_recover_trees+0x628)[0x558782c1aa3b]
./btrfs(+0x83f1f)[0x558782c12f1f]
./btrfs(handle_command_group+0x49)[0x558782ba717b]
./btrfs(main+0x94)[0x558782ba7275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f06496ad7fd]
./btrfs(_start+0x2a)[0x558782ba6e1a]
Aborted

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02 19:51                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
@ 2022-06-02 19:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  2022-06-02 19:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-02 19:53 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, Jun 2, 2022 at 3:51 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Jun 02, 2022 at 03:35:46PM -0400, Josef Bacik wrote:
> > On Thu, Jun 2, 2022 at 3:08 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Thu, Jun 02, 2022 at 02:43:03PM -0400, Josef Bacik wrote:
> > > > Now we run
> > > >
> > > > btrfs rescue tree-recover <device>
> > >
> > > It got pretty far, and then
> > > Invalid mapping for 364837339136-364837355520, got 11106814787584-11107888529408
> > > Couldn't map the block 364837339136
> > > Couldn't map the block 364837339136
> > > deleting slot 0 in block 11160501878784
> > > Invalid mapping for 364837306368-364837322752, got 11106814787584-11107888529408
> > > Couldn't map the block 364837306368
> > > Couldn't map the block 364837306368
> > > deleting slot 0 in block 11160501878784
> > > Invalid mapping for 364746457088-364746473472, got 11106814787584-11107888529408
> > > Couldn't map the block 364746457088
> > > Couldn't map the block 364746457088
> > > deleting slot 0 in block 11160501878784
> >
> > Was it printing a lot of these messages?  I was sort of hoping we
>
> Yes
>
> > found all the chunks so it didn't feel the need to delete a bunch of
> > stuff.  Can you re-run
> >
> > btrfs rescue recover-chunks <device>
> >
> > and make sure it doesn't find anything new?  Maybe there were some
> > system chunks that it found that has the other chunks in it.  Thanks,
>
> output is very long.  I was able to paste a good part in
> https://justpaste.it/5cy5s
>

Ok it seems like we're still missing some chunks, hopefully re-running
btrfs rescue recover-chunks <device> will find the remaining, there
must have been system chunks that got discovered.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02 19:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
@ 2022-06-02 19:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
  2022-06-02 20:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-02 19:56 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Jun 02, 2022 at 03:53:00PM -0400, Josef Bacik wrote:
> Ok it seems like we're still missing some chunks, hopefully re-running
> btrfs rescue recover-chunks <device> will find the remaining, there
> must have been system chunks that got discovered.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
FS_INFO IS 0x55f3efdd3bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55f3efdd3bc0
Walking all our trees and pinning down the currently accessible blocks
Invalid mapping for 11822437826560-11822437842944, got 14271702368256-14272776110080
Couldn't map the block 11822437826560
Couldn't map the block 11822437826560
Error reading root block
ERROR: Couldn't pin down excluded extents, if there were errors run btrfs rescue tree-recover
doing close???
Recover chunks tree failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02 19:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
@ 2022-06-02 20:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  2022-06-02 20:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-02 20:06 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, Jun 2, 2022 at 3:56 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Jun 02, 2022 at 03:53:00PM -0400, Josef Bacik wrote:
> > Ok it seems like we're still missing some chunks, hopefully re-running
> > btrfs rescue recover-chunks <device> will find the remaining, there
> > must have been system chunks that got discovered.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> FS_INFO IS 0x55f3efdd3bc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x55f3efdd3bc0
> Walking all our trees and pinning down the currently accessible blocks
> Invalid mapping for 11822437826560-11822437842944, got 14271702368256-14272776110080
> Couldn't map the block 11822437826560
> Couldn't map the block 11822437826560
> Error reading root block
> ERROR: Couldn't pin down excluded extents, if there were errors run btrfs rescue tree-recover
> doing close???
> Recover chunks tree failed

Pushed, try again please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02 20:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
@ 2022-06-02 20:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  2022-06-03  2:20                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-02 20:32 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Jun 02, 2022 at 04:06:00PM -0400, Josef Bacik wrote:
> On Thu, Jun 2, 2022 at 3:56 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Thu, Jun 02, 2022 at 03:53:00PM -0400, Josef Bacik wrote:
> > > Ok it seems like we're still missing some chunks, hopefully re-running
> > > btrfs rescue recover-chunks <device> will find the remaining, there
> > > must have been system chunks that got discovered.  Thanks,
> >
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> > FS_INFO IS 0x55f3efdd3bc0
> > Couldn't find the last root for 8
> > FS_INFO AFTER IS 0x55f3efdd3bc0
> > Walking all our trees and pinning down the currently accessible blocks
> > Invalid mapping for 11822437826560-11822437842944, got 14271702368256-14272776110080
> > Couldn't map the block 11822437826560
> > Couldn't map the block 11822437826560
> > Error reading root block
> > ERROR: Couldn't pin down excluded extents, if there were errors run btrfs rescue tree-recover
> > doing close???
> > Recover chunks tree failed
> 
> Pushed, try again please.  Thanks,

That worked:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue
recover-chunks /dev/mapper/dshelf1
FS_INFO IS 0x5594c8305bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5594c8305bc0
Walking all our trees and pinning down the currently accessible blocks
No missing chunks, we're all done
doing close???
Recover chunks succeeded, you can run check now


gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x564be0ed2bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x564be0ed2bc0
Checking root 2 bytenr 15645200318464
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645740302336
Checking root 7 bytenr 15645188767744
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645190864896
Checking root 161199 bytenr 15645198614528
Checking root 161200 bytenr 15646048272384
Checking root 161889 bytenr 15645940285440
Checking root 162628 bytenr 15645764812800
Checking root 162632 bytenr 15645196845056
Checking root 163298 bytenr 15645124263936
Checking root 163302 bytenr 15645198188544
Checking root 163303 bytenr 15645199499264
Checking root 163316 bytenr 15645199728640
Checking root 163318 bytenr 15645980491776
scanning, best has 0 found 0 bad
checking block 15645980491776 generation 1597265 fs info generation 2582703
trying bytenr 15645980491776 got 2 blocks 1 bad
Repairing root 163318 bad_blocks 1 update 1
deleting slot 0 in block 15645980491776
bad tree block 15645980491776, invalid nr_items: 0
kernel-shared/disk-io.c:553: write_tree_block: BUG_ON `1` triggered, value 1
./btrfs(+0x25ae7)[0x564be025eae7]
./btrfs(write_tree_block+0xb8)[0x564be0260ec9]
./btrfs(+0x8aa3d)[0x564be02c3a3d]
./btrfs(+0x8add0)[0x564be02c3dd0]
./btrfs(btrfs_recover_trees+0x628)[0x564be02c4a40]
./btrfs(+0x83f24)[0x564be02bcf24]
./btrfs(handle_command_group+0x49)[0x564be025117b]
./btrfs(main+0x94)[0x564be0251275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f55d1ef47fd]
./btrfs(_start+0x2a)[0x564be0250e1a]
Aborted

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-02 20:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
@ 2022-06-03  2:20                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
  2022-06-03 14:47                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-03  2:20 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, Jun 2, 2022 at 4:32 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Jun 02, 2022 at 04:06:00PM -0400, Josef Bacik wrote:
> > On Thu, Jun 2, 2022 at 3:56 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Thu, Jun 02, 2022 at 03:53:00PM -0400, Josef Bacik wrote:
> > > > Ok it seems like we're still missing some chunks, hopefully re-running
> > > > btrfs rescue recover-chunks <device> will find the remaining, there
> > > > must have been system chunks that got discovered.  Thanks,
> > >
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> > > FS_INFO IS 0x55f3efdd3bc0
> > > Couldn't find the last root for 8
> > > FS_INFO AFTER IS 0x55f3efdd3bc0
> > > Walking all our trees and pinning down the currently accessible blocks
> > > Invalid mapping for 11822437826560-11822437842944, got 14271702368256-14272776110080
> > > Couldn't map the block 11822437826560
> > > Couldn't map the block 11822437826560
> > > Error reading root block
> > > ERROR: Couldn't pin down excluded extents, if there were errors run btrfs rescue tree-recover
> > > doing close???
> > > Recover chunks tree failed
> >
> > Pushed, try again please.  Thanks,
>
> That worked:
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue
> recover-chunks /dev/mapper/dshelf1
> FS_INFO IS 0x5594c8305bc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x5594c8305bc0
> Walking all our trees and pinning down the currently accessible blocks
> No missing chunks, we're all done
> doing close???
> Recover chunks succeeded, you can run check now
>
>

Sorry daughters graduation thing took forever, I've updated the code,
it should work now.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-03  2:20                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
@ 2022-06-03 14:47                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  2022-06-03 16:17                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-03 14:47 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Jun 02, 2022 at 10:20:13PM -0400, Josef Bacik wrote:
> Sorry daughters graduation thing took forever, I've updated the code,
> it should work now.  Thanks,

Not sorry, congrats ;)

It works better but seems to be looping on the same thing now (it has
all night):
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-03 14:47                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
@ 2022-06-03 16:17                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
  2022-06-03 16:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-03 16:17 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Fri, Jun 3, 2022 at 10:47 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Jun 02, 2022 at 10:20:13PM -0400, Josef Bacik wrote:
> > Sorry daughters graduation thing took forever, I've updated the code,
> > it should work now.  Thanks,
>
> Not sorry, congrats ;)
>
> It works better but seems to be looping on the same thing now (it has
> all night):

Ok I think I know what it is, try again please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-03 16:17                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
@ 2022-06-03 16:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  2022-06-03 17:07                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-03 16:42 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Fri, Jun 03, 2022 at 12:17:15PM -0400, Josef Bacik wrote:
> On Fri, Jun 3, 2022 at 10:47 AM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Thu, Jun 02, 2022 at 10:20:13PM -0400, Josef Bacik wrote:
> > > Sorry daughters graduation thing took forever, I've updated the code,
> > > it should work now.  Thanks,
> >
> > Not sorry, congrats ;)
> >
> > It works better but seems to be looping on the same thing now (it has
> > all night):
> 
> Ok I think I know what it is, try again please.  Thanks,

Woohoo, that worked:

Invalid mapping for 365033668608-365033684992, got 11106814787584-11107888529408
Couldn't map the block 365033668608
Couldn't map the block 365033668608
deleting slot 0 in block 15646000562176
deleting slot 0 in block 15646000562176
deleting slot 0 in block 15646000562176
Root 164824 was completely cleared, deleting it
Checking root 164825 bytenr 15646001872896
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164825, we're going to clear it and hope for the best
We thought root 164825 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it.
Checking root 164910 bytenr 10194254020608
Invalid mapping for 10194254020608-10194254036992, got 11106814787584-11107888529408
Couldn't map the block 10194254020608
Couldn't map the block 10194254020608
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164910, we're going to clear it and hope for the best
We thought root 164910 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it.
Checking root 164911 bytenr 10194440339456
Invalid mapping for 10194440339456-10194440355840, got 11106814787584-11107888529408
Couldn't map the block 10194440339456
Couldn't map the block 10194440339456
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164911, we're going to clear it and hope for the best
We thought root 164911 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it.
Checking root 164912 bytenr 10194209554432
Invalid mapping for 10194209554432-10194209570816, got 11106814787584-11107888529408
Couldn't map the block 10194209554432
Couldn't map the block 10194209554432
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164912, we're going to clear it and hope for the best
We thought root 164912 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it.
Checking root 164913 bytenr 10194353700864
Invalid mapping for 10194353700864-10194353717248, got 11106814787584-11107888529408
Couldn't map the block 10194353700864
Couldn't map the block 10194353700864
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164913, we're going to clear it and hope for the best
We thought root 164913 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it.
Checking root 164914 bytenr 10194543017984
Invalid mapping for 10194543017984-10194543034368, got 11106814787584-11107888529408
Couldn't map the block 10194543017984
Couldn't map the block 10194543017984
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164914, we're going to clear it and hope for the best
We thought root 164914 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it.
Checking root 164915 bytenr 10678962552832
Invalid mapping for 10678962552832-10678962569216, got 11106814787584-11107888529408
Couldn't map the block 10678962552832
Couldn't map the block 10678962552832
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164915, we're going to clear it and hope for the best
We thought root 164915 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it.
Checking root 164916 bytenr 10679219617792
Invalid mapping for 10679219617792-10679219634176, got 11106814787584-11107888529408
Couldn't map the block 10679219617792
Couldn't map the block 10679219617792
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164916, we're going to clear it and hope for the best
We thought root 164916 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it.
Checking root 164917 bytenr 10678998777856
Invalid mapping for 10678998777856-10678998794240, got 11106814787584-11107888529408
Couldn't map the block 10678998777856
Couldn't map the block 10678998777856
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164917, we're going to clear it and hope for the best
We thought root 164917 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it.
Checking root 18446744073709551607 bytenr 8733327196160
Invalid mapping for 8733327196160-8733327212544, got 11106814787584-11107888529408
Couldn't map the block 8733327196160
Couldn't map the block 8733327196160
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 18446744073709551607, we're going to clear it and hope for the best
We thought root 18446744073709551607 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it.
Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# 

Now running the next step:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x55d3c8ad6bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55d3c8ad6bc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups
deleting space cache for 11106814787584
deleting space cache for 11108962271232
deleting space cache for 11110036013056
deleting space cache for 11111109754880
deleting space cache for 11112183496704
deleting space cache for 11113257238528
deleting space cache for 11114330980352
deleting space cache for 11115404722176
deleting space cache for 11116478464000
deleting space cache for 11118625947648
deleting space cache for 11119699689472
deleting space cache for 11120773431296
deleting space cache for 11121847173120
deleting space cache for 11122920914944
deleting space cache for 11123994656768
deleting space cache for 11125068398592
deleting space cache for 11126142140416
deleting space cache for 11127215882240
deleting space cache for 11128289624064
deleting space cache for 11129363365888
deleting space cache for 11130437107712
deleting space cache for 11131510849536
deleting space cache for 11132584591360
deleting space cache for 11133658333184
(...)


searching 1 for bad extents
processed 81920 of 18446744073709518848 possible bytes, 0%
Found an extent we don't have a block group for in the file
Couldn't find any paths for this inode
Deleting [4483, 108, 0] root 15645018226688 path top 15645018226688 top slot 5 leaf 15645018243072 slot 11

searching 1 for bad extents
processed 81920 of 18446744073709518848 possible bytes, 0%
Found an extent we don't have a block group for in the file
Couldn't find any paths for this inode
Deleting [4484, 108, 0] root 15645018161152 path top 15645018161152 top slot 5 leaf 15645018177536 slot 12

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-03 16:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
@ 2022-06-03 17:07                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-06-03 18:34                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-03 17:07 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Fri, Jun 03, 2022 at 09:42:52AM -0700, Marc MERLIN wrote:
> searching 1 for bad extents
> processed 81920 of 18446744073709518848 possible bytes, 0%
> Found an extent we don't have a block group for in the file
> Couldn't find any paths for this inode
> Deleting [4483, 108, 0] root 15645018226688 path top 15645018226688 top slot 5 leaf 15645018243072 slot 11
> 
> searching 1 for bad extents
> processed 81920 of 18446744073709518848 possible bytes, 0%
> Found an extent we don't have a block group for in the file
> Couldn't find any paths for this inode
> Deleting [4484, 108, 0] root 15645018161152 path top 15645018161152 top slot 5 leaf 15645018177536 slot 12

finished with

searching 164623 for bad extents
processed 278528 of 63193088 possible bytes, 0%
Found an extent we don't have a block group for in the file
ref to path failed
Couldn't find any paths for this inode
Deleting [72784, 108, 0] root 15645019226112 path top 15645019226112 top slot 17 leaf 11160502550528 slot 12

searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
Couldn't find any paths for this inode
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
Deleting [3740, 108, 267370496] root 15645019176960 path top 15645019176960 top slot 0 leaf 15645019553792 slot 0

searching 164624 for bad extents

Found an extent we don't have a block group for in the file
corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
Couldn't find any paths for this inode
corrupt node: root=164624 block=15645019439104 physical=15054973140992 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
ERROR: error searching for key?? -1

wtf
it failed?? -1
ERROR: failed to clear bad extents
doing close???
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=32768
extent buffer leak: start 15645019406336 len 16384
extent buffer leak: start 15645019406336 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019406336 len 16384
extent buffer leak: start 15645019439104 len 16384
extent buffer leak: start 15645019439104 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019439104 len 16384
Init extent tree failed

needs further fixing?
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-03 17:07                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-06-03 18:34                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-06-03 18:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-03 18:34 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Fri, Jun 3, 2022 at 1:07 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Fri, Jun 03, 2022 at 09:42:52AM -0700, Marc MERLIN wrote:
> > searching 1 for bad extents
> > processed 81920 of 18446744073709518848 possible bytes, 0%
> > Found an extent we don't have a block group for in the file
> > Couldn't find any paths for this inode
> > Deleting [4483, 108, 0] root 15645018226688 path top 15645018226688 top slot 5 leaf 15645018243072 slot 11
> >
> > searching 1 for bad extents
> > processed 81920 of 18446744073709518848 possible bytes, 0%
> > Found an extent we don't have a block group for in the file
> > Couldn't find any paths for this inode
> > Deleting [4484, 108, 0] root 15645018161152 path top 15645018161152 top slot 5 leaf 15645018177536 slot 12
>
> finished with
>
> searching 164623 for bad extents
> processed 278528 of 63193088 possible bytes, 0%
> Found an extent we don't have a block group for in the file
> ref to path failed
> Couldn't find any paths for this inode
> Deleting [72784, 108, 0] root 15645019226112 path top 15645019226112 top slot 17 leaf 11160502550528 slot 12
>
> searching 164623 for bad extents
> processed 311296 of 63193088 possible bytes, 0%
> searching 164624 for bad extents
>
> Found an extent we don't have a block group for in the file
> Couldn't find any paths for this inode
> corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0)

Hmm tree-recover is supposed to catch this, can you re-run
tree-recover and see if it finds this block and gets rid of it?
Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-03 18:34                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-06-03 18:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-06-04 12:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-03 18:39 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Fri, Jun 03, 2022 at 02:34:37PM -0400, Josef Bacik wrote:
> Hmm tree-recover is supposed to catch this, can you re-run
> tree-recover and see if it finds this block and gets rid of it?
> Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x55eaff09dbc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55eaff09dbc0
Checking root 2 bytenr 15645019570176
Checking root 4 bytenr 15645019078656h
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645018275840
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645019176960
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
scanning, best has 0 found 0 bad
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
checking block 15645502210048 generation 1601068 fs info generation 2588154
Invalid mapping for 5809703075840-5809703092224, got 11106814787584-11107888529408
Couldn't map the block 5809703075840
Couldn't map the block 5809703075840
Invalid mapping for 4867246325760-4867246342144, got 11106814787584-11107888529408
Couldn't map the block 4867246325760
Couldn't map the block 4867246325760
Invalid mapping for 13577676587008-13577676603392, got 14271702368256-14272776110080
Couldn't map the block 13577676587008
Couldn't map the block 13577676587008
Invalid mapping for 9715195920384-9715195936768, got 11106814787584-11107888529408
Couldn't map the block 9715195920384
Couldn't map the block 9715195920384
Invalid mapping for 9715198033920-9715198050304, got 11106814787584-11107888529408
Couldn't map the block 9715198033920
Couldn't map the block 9715198033920
(...)
Invalid mapping for 13577212362752-13577212379136, got 14271702368256-14272776110080
Couldn't map the block 13577212362752
Couldn't map the block 13577212362752
deleting slot 14 in block 15646005067776
Invalid mapping for 13577154658304-13577154674688, got 14271702368256-14272776110080
Couldn't map the block 13577154658304
Couldn't map the block 13577154658304
deleting slot 14 in block 15646005067776
Invalid mapping for 12512115081216-12512115097600, got 14271702368256-14272776110080
Couldn't map the block 12512115081216
Couldn't map the block 12512115081216
deleting slot 14 in block 15646005067776
Invalid mapping for 11651757277184-11651757293568, got 14271702368256-14272776110080
Couldn't map the block 11651757277184
Couldn't map the block 11651757277184
deleting slot 14 in block 15646005067776
Invalid mapping for 11651852124160-11651852140544, got 14271702368256-14272776110080
Couldn't map the block 11651852124160
Couldn't map the block 11651852124160
deleting slot 15 in block 15646005067776
Invalid mapping for 12511596118016-12511596134400, got 14271702368256-14272776110080
Couldn't map the block 12511596118016
Couldn't map the block 12511596118016
deleting slot 15 in block 15646005067776
Invalid mapping for 10678933913600-10678933929984, got 11106814787584-11107888529408
Couldn't map the block 10678933913600
Couldn't map the block 10678933913600
deleting slot 15 in block 15646005067776
Invalid mapping for 11651771924480-11651771940864, got 14271702368256-14272776110080
Couldn't map the block 11651771924480
Couldn't map the block 11651771924480
deleting slot 15 in block 15646005067776
Invalid mapping for 10194218582016-10194218598400, got 11106814787584-11107888529408
Couldn't map the block 10194218582016
Couldn't map the block 10194218582016
deleting slot 15 in block 15646005067776
Invalid mapping for 13577142059008-13577142075392, got 14271702368256-14272776110080
Couldn't map the block 13577142059008
Couldn't map the block 13577142059008
deleting slot 8 in block 15645502210048
Checking root 164629 bytenr 15645485137920
Checking root 164631 bytenr 15645496983552
Checking root 164633 bytenr 15645526884352
Checking root 164823 bytenr 15645999005696
Tree recovery finished, you can run check now

However I still get:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1

inserting block group 15929559744512
inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
Deleting [260, 108, 0] root 15645019684864 path top 15645019684864 top slot 0 leaf 15645019734016 slot 114

searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
Deleting [260, 108, 58146816] root 15645018718208 path top 15645018718208 top slot 0 leaf 15645019553792 slot 114

searching 164624 for bad extents

Found an extent we don't have a block group for in the file
corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
Couldn't find any paths for this inode
corrupt node: root=164624 block=15645019684864 physical=15054973386752 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
ERROR: error searching for key?? -1

wtf
it failed?? -1
ERROR: failed to clear bad extents
doing close???
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=32768
extent buffer leak: start 15645019504640 len 16384
extent buffer leak: start 15645019504640 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019504640 len 16384
extent buffer leak: start 15645019684864 len 16384
extent buffer leak: start 15645019684864 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019684864 len 16384
Init extent tree failed


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-03 18:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-06-04 12:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-06-04 13:48                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-04 12:49 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Fri, Jun 3, 2022 at 2:39 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Fri, Jun 03, 2022 at 02:34:37PM -0400, Josef Bacik wrote:
> > Hmm tree-recover is supposed to catch this, can you re-run
> > tree-recover and see if it finds this block and gets rid of it?
> > Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> FS_INFO IS 0x55eaff09dbc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x55eaff09dbc0
> Checking root 2 bytenr 15645019570176
> Checking root 4 bytenr 15645019078656h
> Checking root 5 bytenr 15645018161152
> Checking root 7 bytenr 15645018275840
> Checking root 9 bytenr 15645740367872
> Checking root 161197 bytenr 15645018341376
> Checking root 161199 bytenr 15645018652672
> Checking root 161200 bytenr 15645018750976
> Checking root 161889 bytenr 11160502124544
> Checking root 162628 bytenr 15645018931200
> Checking root 162632 bytenr 15645018210304
> Checking root 163298 bytenr 15645019045888
> Checking root 163302 bytenr 15645018685440
> Checking root 163303 bytenr 15645019095040
> Checking root 163316 bytenr 15645018996736
> Checking root 163920 bytenr 15645019144192
> Checking root 164620 bytenr 15645019275264
> Checking root 164623 bytenr 15645019226112
> Checking root 164624 bytenr 15645019176960
> corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
> scanning, best has 0 found 0 bad

Ok we're finding the corrupt blocks and scanning, but for some reason
we're not getting the updated root?

I've pushed a debug patch, can you re-run tree-recover and capture the
output?  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-04 12:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-06-04 13:48                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-06-04 23:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-04 13:48 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sat, Jun 04, 2022 at 08:49:44AM -0400, Josef Bacik wrote:
> Ok we're finding the corrupt blocks and scanning, but for some reason
> we're not getting the updated root?
> 
> I've pushed a debug patch, can you re-run tree-recover and capture the
> output?  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x5570a7eb6bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5570a7eb6bc0
Checking root 2 bytenr 15645019668480
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645019488256
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645018718208
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
scanning, best has 0 found 0 bad
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
checking block 15645018718208 generation 2588157 fs info generation 2588157
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
trying bytenr 15645018718208 got 96 blocks 1 bad
checking block 15645019684864 generation 2588156 fs info generation 2588157
trying bytenr 15645019684864 got 145 blocks 0 bad
checking block 15645502210048 generation 1601068 fs info generation 2588157
trying bytenr 15645502210048 got 146 blocks 0 bad
checking block 15645019471872 generation 2588157 fs info generation 2588157
scan for best root 164624 wants to use 15645502210048 as the root bytenr
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
Repairing root 164624 bad_blocks 0 update 1
setting root 164624 to bytenr 15645502210048
Checking root 164629 bytenr 15645485137920
Checking root 164631 bytenr 15645496983552
Checking root 164633 bytenr 15645526884352
Checking root 164823 bytenr 15645999005696
Tree recovery finished, you can run check now

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-04 13:48                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-06-04 23:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-05  0:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-04 23:10 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sat, Jun 4, 2022 at 9:48 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sat, Jun 04, 2022 at 08:49:44AM -0400, Josef Bacik wrote:
> > Ok we're finding the corrupt blocks and scanning, but for some reason
> > we're not getting the updated root?
> >
> > I've pushed a debug patch, can you re-run tree-recover and capture the
> > output?  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> FS_INFO IS 0x5570a7eb6bc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x5570a7eb6bc0
> Checking root 2 bytenr 15645019668480
> Checking root 4 bytenr 15645019078656
> Checking root 5 bytenr 15645018161152
> Checking root 7 bytenr 15645019488256
> Checking root 9 bytenr 15645740367872
> Checking root 161197 bytenr 15645018341376
> Checking root 161199 bytenr 15645018652672
> Checking root 161200 bytenr 15645018750976
> Checking root 161889 bytenr 11160502124544
> Checking root 162628 bytenr 15645018931200
> Checking root 162632 bytenr 15645018210304
> Checking root 163298 bytenr 15645019045888
> Checking root 163302 bytenr 15645018685440
> Checking root 163303 bytenr 15645019095040
> Checking root 163316 bytenr 15645018996736
> Checking root 163920 bytenr 15645019144192
> Checking root 164620 bytenr 15645019275264
> Checking root 164623 bytenr 15645019226112
> Checking root 164624 bytenr 15645018718208
> corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> scanning, best has 0 found 0 bad
> corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> checking block 15645018718208 generation 2588157 fs info generation 2588157
> corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> trying bytenr 15645018718208 got 96 blocks 1 bad
> checking block 15645019684864 generation 2588156 fs info generation 2588157
> trying bytenr 15645019684864 got 145 blocks 0 bad
> checking block 15645502210048 generation 1601068 fs info generation 2588157
> trying bytenr 15645502210048 got 146 blocks 0 bad
> checking block 15645019471872 generation 2588157 fs info generation 2588157
> scan for best root 164624 wants to use 15645502210048 as the root bytenr
> corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> Repairing root 164624 bad_blocks 0 update 1
> setting root 164624 to bytenr 15645502210048

Ok this looks like it worked?  Can you re-run tree-recover to see if
it uses the right bytenr?  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-04 23:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-06-05  0:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-05 19:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-05  0:13 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sat, Jun 04, 2022 at 07:10:16PM -0400, Josef Bacik wrote:
 
> Ok this looks like it worked?  Can you re-run tree-recover to see if
> it uses the right bytenr?  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x5568318c7bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5568318c7bc0
Checking root 2 bytenr 15645019668480
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645019488256
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645502210048
Checking root 164629 bytenr 15645485137920
Checking root 164631 bytenr 15645496983552
Checking root 164633 bytenr 15645526884352
Checking root 164823 bytenr 15645999005696
Tree recovery finished, you can run check now

But
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1           
(...)
nserting block group 15920969809920
inserting block group 15922043551744
inserting block group 15923117293568
inserting block group 15924191035392
inserting block group 15925264777216
inserting block group 15926338519040
inserting block group 15927412260864
inserting block group 15928486002688
inserting block group 15929559744512
inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
Deleting [260, 108, 0] root 15645019553792 path top 15645019553792 top slot 0 leaf 15645019602944 slot 114

searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
Deleting [260, 108, 58146816] root 15645018226688 path top 15645018226688 top slot 0 leaf 15645018718208 slot 114

searching 164624 for bad extents

Found an extent we don't have a block group for in the file
corrupt node: root=164624 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
Couldn't find any paths for this inode
corrupt node: root=164624 block=15645019553792 physical=15054973255680 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
ERROR: error searching for key?? -1

wtf
it failed?? -1
ERROR: failed to clear bad extents
doing close???
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=32768
extent buffer leak: start 15645018374144 len 16384
extent buffer leak: start 15645018374144 len 16384
WARNING: dirty eb leak (aborted trans): start 15645018374144 len 16384
extent buffer leak: start 15645019553792 len 16384
extent buffer leak: start 15645019553792 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019553792 len 16384
Init extent tree failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-05  0:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-05 19:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-06-05 20:11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-05 19:37 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sat, Jun 4, 2022 at 8:13 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sat, Jun 04, 2022 at 07:10:16PM -0400, Josef Bacik wrote:
>
> > Ok this looks like it worked?  Can you re-run tree-recover to see if
> > it uses the right bytenr?  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1

Sorry I've been in the mountains with terrible internet.  I'm home
with a computer and real internet now.  I've pushed more debugging,
just run the init-extent-tree again, it'll spit out the root info so I
can see wtf is going on here.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-05 19:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-06-05 20:11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-06-05 20:58                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-05 20:11 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Jun 05, 2022 at 03:37:23PM -0400, Josef Bacik wrote:
> On Sat, Jun 4, 2022 at 8:13 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sat, Jun 04, 2022 at 07:10:16PM -0400, Josef Bacik wrote:
> >
> > > Ok this looks like it worked?  Can you re-run tree-recover to see if
> > > it uses the right bytenr?  Thanks,
> >
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> 
> Sorry I've been in the mountains with terrible internet.  I'm home
> with a computer and real internet now.  I've pushed more debugging,
> just run the init-extent-tree again, it'll spit out the root info so I
> can see wtf is going on here.  Thanks,

Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x555555650bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555650bc0
Walking all our trees and pinning down the currently accessible blocks

Program received signal SIGSEGV, Segmentation fault.
0x000055555557103f in generic_err (buf=buf@entry=0x5555557a3bb0, slot=slot@entry=38, 
    fmt=fmt@entry=0x55555560be80 "bad key order, current (%llu %u %llu) next (%llu %u %llu)") at ./kernel-shared/ctree.h:2135
2135    BTRFS_SETGET_HEADER_FUNCS(header_bytenr, struct btrfs_header, bytenr, 64);
(gdb) bt
#0  0x000055555557103f in generic_err (buf=buf@entry=0x5555557a3bb0, slot=slot@entry=38, 
    fmt=fmt@entry=0x55555560be80 "bad key order, current (%llu %u %llu) next (%llu %u %llu)") at ./kernel-shared/ctree.h:2135
#1  0x000055555557303e in btrfs_check_node (fs_info=fs_info@entry=0x555555650bc0, parent_key=parent_key@entry=0x0, 
    node=node@entry=0x5555557a3bb0) at kernel-shared/ctree.c:663
#2  0x000055555557c0db in read_tree_block (fs_info=fs_info@entry=0x555555650bc0, bytenr=<optimized out>, 
    parent_transid=parent_transid@entry=0) at kernel-shared/disk-io.c:403
#3  0x000055555559f54e in traverse_tree_blocks (tree=tree@entry=0x555555701690, eb=eb@entry=0x55555579fb30, 
    tree_root=tree_root@entry=0) at common/repair.c:166
#4  0x000055555559f4ca in traverse_tree_blocks (tree=tree@entry=0x555555701690, eb=eb@entry=0x5555557938f0, 
    tree_root=tree_root@entry=1) at common/repair.c:146
#5  0x000055555559f5b5 in traverse_tree_blocks (tree=tree@entry=0x555555701690, eb=0x5555556a2be0, tree_root=tree_root@entry=1)
    at common/repair.c:171
#6  0x000055555559f7e5 in btrfs_mark_used_tree_blocks (fs_info=fs_info@entry=0x555555650bc0, tree=tree@entry=0x555555701690, 
    chunk_only=chunk_only@entry=false) at common/repair.c:189
#7  0x00005555555e11ed in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:1211
#8  0x00005555555d7eb4 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:139
#9  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555646c80 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#10 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#11 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555647cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#12 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-05 20:11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-06-05 20:58                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-06-05 21:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-05 20:58 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Jun 5, 2022 at 4:11 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, Jun 05, 2022 at 03:37:23PM -0400, Josef Bacik wrote:
> > On Sat, Jun 4, 2022 at 8:13 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sat, Jun 04, 2022 at 07:10:16PM -0400, Josef Bacik wrote:
> > >
> > > > Ok this looks like it worked?  Can you re-run tree-recover to see if
> > > > it uses the right bytenr?  Thanks,
> > >
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> >
> > Sorry I've been in the mountains with terrible internet.  I'm home
> > with a computer and real internet now.  I've pushed more debugging,
> > just run the init-extent-tree again, it'll spit out the root info so I
> > can see wtf is going on here.  Thanks,
>
> Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> FS_INFO IS 0x555555650bc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x555555650bc0
> Walking all our trees and pinning down the currently accessible blocks
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x000055555557103f in generic_err (buf=buf@entry=0x5555557a3bb0, slot=slot@entry=38,
>     fmt=fmt@entry=0x55555560be80 "bad key order, current (%llu %u %llu) next (%llu %u %llu)") at ./kernel-shared/ctree.h:2135
> 2135    BTRFS_SETGET_HEADER_FUNCS(header_bytenr, struct btrfs_header, bytenr, 64);
> (gdb) bt

Sigh try again please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-05 20:58                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-06-05 21:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-06-05 21:43                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-05 21:26 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Jun 05, 2022 at 04:58:15PM -0400, Josef Bacik wrote:
> 
> Sigh try again please.  Thanks,

Same

FS_INFO IS 0x555555650bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555650bc0
Walking all our trees and pinning down the currently accessible blocks

Program received signal SIGSEGV, Segmentation fault.
generic_err (buf=buf@entry=0x5555557a3bb0, slot=slot@entry=38, 
    fmt=fmt@entry=0x55555560be80 "bad key order, current (%llu %u %llu) next (%llu %u %llu)") at kernel-shared/ctree.c:605
605                             start = root->node ? root->node->start : 0;
(gdb) bt
#0  generic_err (buf=buf@entry=0x5555557a3bb0, slot=slot@entry=38, 
    fmt=fmt@entry=0x55555560be80 "bad key order, current (%llu %u %llu) next (%llu %u %llu)") at kernel-shared/ctree.c:605
#1  0x0000555555573065 in btrfs_check_node (fs_info=fs_info@entry=0x555555650bc0, parent_key=parent_key@entry=0x0, 
    node=node@entry=0x5555557a3bb0) at kernel-shared/ctree.c:673
#2  0x000055555557c102 in read_tree_block (fs_info=fs_info@entry=0x555555650bc0, bytenr=<optimized out>, 
    parent_transid=parent_transid@entry=0) at kernel-shared/disk-io.c:403
#3  0x000055555559f575 in traverse_tree_blocks (tree=tree@entry=0x555555701690, eb=eb@entry=0x55555579fb30, 
    tree_root=tree_root@entry=0) at common/repair.c:166
#4  0x000055555559f4f1 in traverse_tree_blocks (tree=tree@entry=0x555555701690, eb=eb@entry=0x5555557938f0, 
    tree_root=tree_root@entry=1) at common/repair.c:146
#5  0x000055555559f5dc in traverse_tree_blocks (tree=tree@entry=0x555555701690, eb=0x5555556a2be0, tree_root=tree_root@entry=1)
    at common/repair.c:171
#6  0x000055555559f80c in btrfs_mark_used_tree_blocks (fs_info=fs_info@entry=0x555555650bc0, tree=tree@entry=0x555555701690, 
    chunk_only=chunk_only@entry=false) at common/repair.c:189
#7  0x00005555555e1214 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:1211
#8  0x00005555555d7edb in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:139
#9  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555646c80 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#10 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#11 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555647cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#12 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-05 21:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-06-05 21:43                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-06-05 21:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-05 21:43 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Jun 5, 2022 at 5:26 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, Jun 05, 2022 at 04:58:15PM -0400, Josef Bacik wrote:
> >
> > Sigh try again please.  Thanks,
>
> Same
>

Sorry, this one should work.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-05 21:43                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-06-05 21:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-06-05 23:03                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-05 21:50 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Jun 05, 2022 at 05:43:41PM -0400, Josef Bacik wrote:
> On Sun, Jun 5, 2022 at 5:26 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sun, Jun 05, 2022 at 04:58:15PM -0400, Josef Bacik wrote:
> > >
> > > Sigh try again please.  Thanks,
> >
> > Same
> 
> Sorry, this one should work.  Thanks,

FS_INFO IS 0x555555650bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555650bc0
Walking all our trees and pinning down the currently accessible blocks
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
Clearing the extent root and re-init'ing the block groups
deleting space cache for 11106814787584
deleting space cache for 11108962271232
deleting space cache for 11110036013056
deleting space cache for 11111109754880
deleting space cache for 11112183496704
(...)
inserting block group 15929559744512
inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
corrupt node: root=164624 root bytenr 15645018226688 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
Couldn't find any paths for this inode
corrupt node: root=164624 root bytenr 15645019652096 commit bytenr 15645018226688 block=15645019668480 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
ERROR: error searching for key?? -1 root 164624 node 15645019652096 commit 15645018226688

wtf
it failed?? -1
ERROR: failed to clear bad extents
doing close???
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=32768
extent buffer leak: start 15645019652096 len 16384
extent buffer leak: start 15645019652096 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019652096 len 16384
extent buffer leak: start 15645019668480 len 16384
extent buffer leak: start 15645019668480 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019668480 len 16384
Init extent tree failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-05 21:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-06-05 23:03                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-06  0:05                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-05 23:03 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Jun 5, 2022 at 5:50 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, Jun 05, 2022 at 05:43:41PM -0400, Josef Bacik wrote:
> > On Sun, Jun 5, 2022 at 5:26 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sun, Jun 05, 2022 at 04:58:15PM -0400, Josef Bacik wrote:
> > > >
> > > > Sigh try again please.  Thanks,
> > >
> > > Same
> >
> > Sorry, this one should work.  Thanks,
>
> FS_INFO IS 0x555555650bc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x555555650bc0
> Walking all our trees and pinning down the currently accessible blocks
> corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> Clearing the extent root and re-init'ing the block groups
> deleting space cache for 11106814787584
> deleting space cache for 11108962271232
> deleting space cache for 11110036013056
> deleting space cache for 11111109754880
> deleting space cache for 11112183496704
> (...)
> inserting block group 15929559744512
> inserting block group 15930633486336
> inserting block group 15931707228160
> inserting block group 15932780969984
> inserting block group 15933854711808
> ERROR: Error reading data reloc tree -2
>
> ERROR: failed to reinit the data reloc root
> searching 1 for bad extents
> processed 999424 of 0 possible bytes, 0%
> searching 4 for bad extents
> processed 163840 of 1064960 possible bytes, 15%
> searching 5 for bad extents
> processed 65536 of 10960896 possible bytes, 0%
> searching 7 for bad extents
> processed 16384 of 16570974208 possible bytes, 0%
> searching 9 for bad extents
> processed 16384 of 16384 possible bytes, 100%
> searching 161197 for bad extents
> processed 131072 of 108986368 possible bytes, 0%
> searching 161199 for bad extents
> processed 196608 of 49479680 possible bytes, 0%
> searching 161200 for bad extents
> processed 180224 of 254214144 possible bytes, 0%
> searching 161889 for bad extents
> processed 229376 of 49446912 possible bytes, 0%
> searching 162628 for bad extents
> processed 49152 of 49463296 possible bytes, 0%
> searching 162632 for bad extents
> processed 147456 of 94633984 possible bytes, 0%
> searching 163298 for bad extents
> processed 49152 of 49463296 possible bytes, 0%
> searching 163302 for bad extents
> processed 147456 of 94633984 possible bytes, 0%
> searching 163303 for bad extents
> processed 131072 of 76333056 possible bytes, 0%
> searching 163316 for bad extents
> processed 147456 of 108544000 possible bytes, 0%
> searching 163920 for bad extents
> processed 16384 of 108691456 possible bytes, 0%
> searching 164620 for bad extents
> processed 49152 of 49463296 possible bytes, 0%
> searching 164623 for bad extents
> processed 311296 of 63193088 possible bytes, 0%
> searching 164624 for bad extents
>
> Found an extent we don't have a block group for in the file
> corrupt node: root=164624 root bytenr 15645018226688 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> Couldn't find any paths for this inode
> corrupt node: root=164624 root bytenr 15645019652096 commit bytenr 15645018226688 block=15645019668480 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> ERROR: error searching for key?? -1 root 164624 node 15645019652096 commit 15645018226688
>

I wonder if our delete thing is corrupting stuff.  Can you re-run
tree-recover, and then once that's done run init-extent-tree?  I put
some stuff to check block all the time to see if we're introducing the
problem.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-05 23:03                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-06-06  0:05                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-06  1:11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-06  0:05 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Jun 05, 2022 at 07:03:31PM -0400, Josef Bacik wrote:
> I wonder if our delete thing is corrupting stuff.  Can you re-run
> tree-recover, and then once that's done run init-extent-tree?  I put
> some stuff to check block all the time to see if we're introducing the
> problem.  Thanks,



gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x5637d77f4bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5637d77f4bc0
Checking root 2 bytenr 15645019471872
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645018587136
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645018226688
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
scanning, best has 0 found 0 bad
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
trying bytenr 15645018226688 got 96 blocks 1 bad 
checking block 15645502210048 generation 1601068 fs info generation 2588162
trying bytenr 15645502210048 got 146 blocks 0 bad 
checking block 15645018324992 generation 2588160 fs info generation 2588162
scan for best root 164624 wants to use 15645502210048 as the root bytenr
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
Repairing root 164624 bad_blocks 0 update 1
setting root 164624 to bytenr 15645502210048
Checking root 164629 bytenr 15645485137920
Checking root 164631 bytenr 15645496983552
Checking root 164633 bytenr 15645526884352
Checking root 164823 bytenr 15645999005696
Tree recovery finished, you can run check now


gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
FS_INFO IS 0x55d4f966dbc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55d4f966dbc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups
deleting space cache for 11106814787584
deleting space cache for 11108962271232
deleting space cache for 11110036013056 
deleting space cache for 11111109754880
deleting space cache for 11112183496704 
deleting space cache for 11113257238528
deleting space cache for 11114330980352
deleting space cache for 11115404722176
deleting space cache for 11116478464000
(...)
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
Deleting [260, 108, 0] root 15645019602944 path top 15645019602944 top slot 0 leaf 15645019701248 slot 114

searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 root bytenr 15645018226688 commit bytenr 15645019602944 block=15645019176960 physical=15054972878848 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
Deleting [260, 108, 58146816] root 15645018226688 path top 15645018226688 top slot 0 leaf 15645019537408 slot 114
corrupt node: root=164624 root bytenr 15645018226688 commit bytenr 15645019602944 block=15645019176960 physical=15054972878848 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
kernel-shared/ctree.c:118: btrfs_release_path: BUG_ON `ret` triggered, value -5
./btrfs(btrfs_release_path+0x62)[0x55d4f7cd950d]
./btrfs(+0x8c751)[0x55d4f7d46751]
./btrfs(+0x8c3c1)[0x55d4f7d463c1]
./btrfs(+0x8c3c1)[0x55d4f7d463c1]
./btrfs(+0x8c8fc)[0x55d4f7d468fc]
./btrfs(+0x8c1cb)[0x55d4f7d461cb]
./btrfs(btrfs_init_extent_tree+0xc83)[0x55d4f7d47e94]
./btrfs(+0x83f6a)[0x55d4f7d3df6a]
./btrfs(handle_command_group+0x49)[0x55d4f7cd217b]
./btrfs(main+0x94)[0x55d4f7cd2275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f7d302a07fd]
./btrfs(_start+0x2a)[0x55d4f7cd1e1a]
Aborted



-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-06  0:05                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-06  1:11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-06-06  1:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-06  1:11 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Jun 5, 2022 at 8:05 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, Jun 05, 2022 at 07:03:31PM -0400, Josef Bacik wrote:
> > I wonder if our delete thing is corrupting stuff.  Can you re-run
> > tree-recover, and then once that's done run init-extent-tree?  I put
> > some stuff to check block all the time to see if we're introducing the
> > problem.  Thanks,
>
>
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1

Ok more targeted debugging to figure out where the problem is coming
from specifically, but hooray I was right.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-06  1:11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-06-06  1:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-06-06 20:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-06  1:22 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Jun 05, 2022 at 09:11:28PM -0400, Josef Bacik wrote:
> On Sun, Jun 5, 2022 at 8:05 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sun, Jun 05, 2022 at 07:03:31PM -0400, Josef Bacik wrote:
> > > I wonder if our delete thing is corrupting stuff.  Can you re-run
> > > tree-recover, and then once that's done run init-extent-tree?  I put
> > > some stuff to check block all the time to see if we're introducing the
> > > problem.  Thanks,
> >
> >
> >
> > gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1
> 
> Ok more targeted debugging to figure out where the problem is coming
> from specifically, but hooray I was right.  Thanks,

searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
cmds/rescue-init-extent-tree.c:197: delete_item: BUG_ON `check_path(&path)` triggered, value -5
./btrfs(+0x8cb28)[0x556144725b28]
./btrfs(+0x8c63d)[0x55614472563d]
./btrfs(+0x8c63d)[0x55614472563d]
./btrfs(+0x8cc2c)[0x556144725c2c]
./btrfs(+0x8c437)[0x556144725437]
./btrfs(btrfs_init_extent_tree+0xc83)[0x5561447271c4]
./btrfs(+0x841d6)[0x55614471d1d6]
./btrfs(handle_command_group+0x49)[0x5561446b117b]
./btrfs(main+0x94)[0x5561446b1275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fea50bb37fd]
./btrfs(_start+0x2a)[0x5561446b0e1a]
Aborted

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-06  1:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-06-06 20:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-06-06 21:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-06 20:42 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Jun 5, 2022 at 9:22 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, Jun 05, 2022 at 09:11:28PM -0400, Josef Bacik wrote:
> > On Sun, Jun 5, 2022 at 8:05 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sun, Jun 05, 2022 at 07:03:31PM -0400, Josef Bacik wrote:
> > > > I wonder if our delete thing is corrupting stuff.  Can you re-run
> > > > tree-recover, and then once that's done run init-extent-tree?  I put
> > > > some stuff to check block all the time to see if we're introducing the
> > > > problem.  Thanks,
> > >
> > >
> > >
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1
> >
> > Ok more targeted debugging to figure out where the problem is coming
> > from specifically, but hooray I was right.  Thanks,
>
> searching 164623 for bad extents
> processed 311296 of 63193088 possible bytes, 0%
> searching 164624 for bad extents
>
> Found an extent we don't have a block group for in the file
> file
> corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> cmds/rescue-init-extent-tree.c:197: delete_item: BUG_ON `check_path(&path)` triggered, value -5

Cool, must be in balance, lets try this again.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-06 20:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-06-06 21:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-06-06 21:19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-06 21:08 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Jun 06, 2022 at 04:42:30PM -0400, Josef Bacik wrote:
> On Sun, Jun 5, 2022 at 9:22 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Sun, Jun 05, 2022 at 09:11:28PM -0400, Josef Bacik wrote:
> > > On Sun, Jun 5, 2022 at 8:05 PM Marc MERLIN <marc@merlins.org> wrote:
> > > >
> > > > On Sun, Jun 05, 2022 at 07:03:31PM -0400, Josef Bacik wrote:
> > > > > I wonder if our delete thing is corrupting stuff.  Can you re-run
> > > > > tree-recover, and then once that's done run init-extent-tree?  I put
> > > > > some stuff to check block all the time to see if we're introducing the
> > > > > problem.  Thanks,
> > > >
> > > >
> > > >
> > > > gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1
> > >
> > > Ok more targeted debugging to figure out where the problem is coming
> > > from specifically, but hooray I was right.  Thanks,
> >
> > searching 164623 for bad extents
> > processed 311296 of 63193088 possible bytes, 0%
> > searching 164624 for bad extents
> >
> > Found an extent we don't have a block group for in the file
> > file
> > corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> > corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> > cmds/rescue-init-extent-tree.c:197: delete_item: BUG_ON `check_path(&path)` triggered, value -5
> 
> Cool, must be in balance, lets try this again.  Thanks,

Same?

inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
kernel-shared/ctree.c:1100: balance_level: BUG_ON `check_path(path)` triggered, value -5
./btrfs(+0x1cf51)[0x564087a36f51]
./btrfs(btrfs_search_slot+0x11cd)[0x564087a3ca10]
./btrfs(+0x8caa5)[0x564087aa6aa5]
./btrfs(+0x8c7d4)[0x564087aa67d4]
./btrfs(+0x8c7d4)[0x564087aa67d4]
./btrfs(+0x8cdc3)[0x564087aa6dc3]
./btrfs(+0x8c5ce)[0x564087aa65ce]
./btrfs(btrfs_init_extent_tree+0xc83)[0x564087aa835b]
./btrfs(+0x8436d)[0x564087a9e36d]
./btrfs(handle_command_group+0x49)[0x564087a3217b]
./btrfs(main+0x94)[0x564087a32275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fb85b01c7fd]
./btrfs(_start+0x2a)[0x564087a31e1a]
Aborted

gargamel:/var/local/src/btrfs-progs-josefbacik# git pull
Already up-to-date.

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-06 21:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-06-06 21:19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-06-06 21:23                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-06 21:19 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Jun 6, 2022 at 5:08 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Jun 06, 2022 at 04:42:30PM -0400, Josef Bacik wrote:
> > On Sun, Jun 5, 2022 at 9:22 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Sun, Jun 05, 2022 at 09:11:28PM -0400, Josef Bacik wrote:
> > > > On Sun, Jun 5, 2022 at 8:05 PM Marc MERLIN <marc@merlins.org> wrote:
> > > > >
> > > > > On Sun, Jun 05, 2022 at 07:03:31PM -0400, Josef Bacik wrote:
> > > > > > I wonder if our delete thing is corrupting stuff.  Can you re-run
> > > > > > tree-recover, and then once that's done run init-extent-tree?  I put
> > > > > > some stuff to check block all the time to see if we're introducing the
> > > > > > problem.  Thanks,
> > > > >
> > > > >
> > > > >
> > > > > gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1
> > > >
> > > > Ok more targeted debugging to figure out where the problem is coming
> > > > from specifically, but hooray I was right.  Thanks,
> > >
> > > searching 164623 for bad extents
> > > processed 311296 of 63193088 possible bytes, 0%
> > > searching 164624 for bad extents
> > >
> > > Found an extent we don't have a block group for in the file
> > > file
> > > corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> > > corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> > > cmds/rescue-init-extent-tree.c:197: delete_item: BUG_ON `check_path(&path)` triggered, value -5
> >
> > Cool, must be in balance, lets try this again.  Thanks,
>
> Same?
>

Nope different spot, I added some more printf's to narrow down which
path is messing up the key order.  Thanks,

josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-06 21:19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-06-06 21:23                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-06-06 21:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-06 21:23 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Jun 06, 2022 at 05:19:40PM -0400, Josef Bacik wrote:
> Nope different spot, I added some more printf's to narrow down which
> path is messing up the key order.  Thanks,

processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
Flying/Flying Wild Alaska/Flying Wild Alaska - 02x04  Era Alaska Rises Again - 624x352 - 1012kbps - xvid.avi
push node left from right mid nritems 48 right nritems 0
setting parent slot 0 to [256 1 0]
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
kernel-shared/ctree.c:1110: balance_level: BUG_ON `check_path(path)` triggered, value -5
./btrfs(+0x1cf51)[0x557391728f51]
./btrfs(btrfs_search_slot+0x12bc)[0x55739172eaff]
./btrfs(+0x8cb9c)[0x557391798b9c]
./btrfs(+0x8c8cb)[0x5573917988cb]
./btrfs(+0x8c8cb)[0x5573917988cb]
./btrfs(+0x8ceba)[0x557391798eba]
./btrfs(+0x8c6c5)[0x5573917986c5]
./btrfs(btrfs_init_extent_tree+0xc83)[0x55739179a452]
./btrfs(+0x84464)[0x557391790464]
./btrfs(handle_command_group+0x49)[0x55739172417b]
./btrfs(main+0x94)[0x557391724275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f669efba7fd]
./btrfs(_start+0x2a)[0x557391723e1a]
Aborted

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-06 21:23                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-06-06 21:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-06 21:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-06 21:39 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Jun 6, 2022 at 5:23 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Jun 06, 2022 at 05:19:40PM -0400, Josef Bacik wrote:
> > Nope different spot, I added some more printf's to narrow down which
> > path is messing up the key order.  Thanks,
>
> processed 49152 of 49463296 possible bytes, 0%
> searching 164623 for bad extents
> processed 311296 of 63193088 possible bytes, 0%
> searching 164624 for bad extents
>
> Found an extent we don't have a block group for in the file
> Flying/Flying Wild Alaska/Flying Wild Alaska - 02x04  Era Alaska Rises Again - 624x352 - 1012kbps - xvid.avi
> push node left from right mid nritems 48 right nritems 0
> setting parent slot 0 to [256 1 0]
> corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)

Hmm, this sounds like we're not adjusting nritems, but the code
definitely is, so I'm sort of nervous about what's going to be
uncovered here.  Added some more information,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-06 21:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-06-06 21:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-06 22:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-06 21:50 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Jun 06, 2022 at 05:39:45PM -0400, Josef Bacik wrote:
> On Mon, Jun 6, 2022 at 5:23 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Mon, Jun 06, 2022 at 05:19:40PM -0400, Josef Bacik wrote:
> > > Nope different spot, I added some more printf's to narrow down which
> > > path is messing up the key order.  Thanks,
> >
> > processed 49152 of 49463296 possible bytes, 0%
> > searching 164623 for bad extents
> > processed 311296 of 63193088 possible bytes, 0%
> > searching 164624 for bad extents
> >
> > Found an extent we don't have a block group for in the file
> > file
> > push node left from right mid nritems 48 right nritems 0
> > setting parent slot 0 to [256 1 0]
> > corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> 
> Hmm, this sounds like we're not adjusting nritems, but the code
> definitely is, so I'm sort of nervous about what's going to be
> uncovered here.  Added some more information,

processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
push node left from right mid nritems 48 right nritems 0 parent 15645019684864 parent nritems 7
parent nritems is now 6
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
kernel-shared/ctree.c:1042: balance_level: BUG_ON `check_path(path)` triggered, value -5
./btrfs(+0x1cf51)[0x557f0d136f51]
./btrfs(btrfs_search_slot+0x102a)[0x557f0d13c86d]
./btrfs(+0x8cc50)[0x557f0d1a6c50]
./btrfs(+0x8c97f)[0x557f0d1a697f]
./btrfs(+0x8c97f)[0x557f0d1a697f]
./btrfs(+0x8cf6e)[0x557f0d1a6f6e]
./btrfs(+0x8c779)[0x557f0d1a6779]
./btrfs(btrfs_init_extent_tree+0xc83)[0x557f0d1a8506]
./btrfs(+0x84518)[0x557f0d19e518]
./btrfs(handle_command_group+0x49)[0x557f0d13217b]
./btrfs(main+0x94)[0x557f0d132275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f54f44bb7fd]
./btrfs(_start+0x2a)[0x557f0d131e1a]
Aborted

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-06 21:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-06 22:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-06-06 22:17                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-06 22:00 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Jun 6, 2022 at 5:50 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Jun 06, 2022 at 05:39:45PM -0400, Josef Bacik wrote:
> > On Mon, Jun 6, 2022 at 5:23 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Mon, Jun 06, 2022 at 05:19:40PM -0400, Josef Bacik wrote:
> > > > Nope different spot, I added some more printf's to narrow down which
> > > > path is messing up the key order.  Thanks,
> > >
> > > processed 49152 of 49463296 possible bytes, 0%
> > > searching 164623 for bad extents
> > > processed 311296 of 63193088 possible bytes, 0%
> > > searching 164624 for bad extents
> > >
> > > Found an extent we don't have a block group for in the file
> > > file
> > > push node left from right mid nritems 48 right nritems 0
> > > setting parent slot 0 to [256 1 0]
> > > corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> >
> > Hmm, this sounds like we're not adjusting nritems, but the code
> > definitely is, so I'm sort of nervous about what's going to be
> > uncovered here.  Added some more information,
>
> processed 49152 of 49463296 possible bytes, 0%
> searching 164623 for bad extents
> processed 311296 of 63193088 possible bytes, 0%
> searching 164624 for bad extents
>
> Found an extent we don't have a block group for in the file
> file
> push node left from right mid nritems 48 right nritems 0 parent 15645019684864 parent nritems 7
> parent nritems is now 6
> corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> kernel-shared/ctree.c:1042: balance_level: BUG_ON `check_path(path)` triggered, value -5

Hmm ok this may just be a new thing I have to check for in
tree-recover.  Give this a shot please, thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-06 22:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-06-06 22:17                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-06-07  2:28                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-06 22:17 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Jun 06, 2022 at 06:00:10PM -0400, Josef Bacik wrote:
> > Found an extent we don't have a block group for in the file
> > file
> > push node left from right mid nritems 48 right nritems 0 parent 15645019684864 parent nritems 7
> > parent nritems is now 6
> > corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> > kernel-shared/ctree.c:1042: balance_level: BUG_ON `check_path(path)` triggered, value -5
> 
> Hmm ok this may just be a new thing I have to check for in
> tree-recover.  Give this a shot please, thanks,

searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
kernel-shared/ctree.c:1718: push_node_left: BUG_ON `check_sibling_keys(dst, src)` triggered, value 1
./btrfs(+0x1cf51)[0x5597970f2f51]
./btrfs(+0x1ef96)[0x5597970f4f96]
./btrfs(btrfs_search_slot+0xee4)[0x5597970f8881]
./btrfs(+0x8cdb6)[0x559797162db6]
./btrfs(+0x8cae5)[0x559797162ae5]
./btrfs(+0x8cae5)[0x559797162ae5]
./btrfs(+0x8d0d4)[0x5597971630d4]
./btrfs(+0x8c8df)[0x5597971628df]
./btrfs(btrfs_init_extent_tree+0xc83)[0x55979716466c]
./btrfs(+0x8467e)[0x55979715a67e]
./btrfs(handle_command_group+0x49)[0x5597970ee17b]
./btrfs(main+0x94)[0x5597970ee275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f6b6f4567fd]
./btrfs(_start+0x2a)[0x5597970ede1a]
Aborted

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-06 22:17                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-06-07  2:28                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-06-07  2:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-07  2:28 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Jun 6, 2022 at 6:17 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Jun 06, 2022 at 06:00:10PM -0400, Josef Bacik wrote:
> > > Found an extent we don't have a block group for in the file
> > > file
> > > push node left from right mid nritems 48 right nritems 0 parent 15645019684864 parent nritems 7
> > > parent nritems is now 6
> > > corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
> > > kernel-shared/ctree.c:1042: balance_level: BUG_ON `check_path(path)` triggered, value -5
> >
> > Hmm ok this may just be a new thing I have to check for in
> > tree-recover.  Give this a shot please, thanks,
>
> searching 164623 for bad extents
> processed 311296 of 63193088 possible bytes, 0%
> searching 164624 for bad extents
>
> Found an extent we don't have a block group for in the file
> file

Ok I thought I caught this particular problem but I don't, so I fixed
tree-recover to handle unordered keys in different nodes.  Pull and
build, run tree-recover.  It's going to start deleting slots for
unordered keys, picking whichever node is newer as the source of
truth.  You should only see this happen on 164624, if you see it fire
a bunch right away stop it and send me the output so I can make sure I
didn't screw anything up.  I went over the code and diff a few times
to make sure I didn't mess anything up, but I could have missed
something.  If that runs and fixes stuff, run it again just to make
sure it doesn't find anything the second time.  It shouldn't since I
re-start the loop if we adjust things, but just in case.  I assume
this will blow up, but if it doesn't you can try running
init-extent-tree again and see how that goes.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07  2:28                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-06-07  2:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-06-07  2:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-07  2:37 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Jun 06, 2022 at 10:28:25PM -0400, Josef Bacik wrote:
> Ok I thought I caught this particular problem but I don't, so I fixed
> tree-recover to handle unordered keys in different nodes.  Pull and
> build, run tree-recover.  It's going to start deleting slots for
> unordered keys, picking whichever node is newer as the source of
> truth.  You should only see this happen on 164624, if you see it fire
> a bunch right away stop it and send me the output so I can make sure I
> didn't screw anything up.  I went over the code and diff a few times
> to make sure I didn't mess anything up, but I could have missed
> something.  If that runs and fixes stuff, run it again just to make
> sure it doesn't find anything the second time.  It shouldn't since I
> re-start the loop if we adjust things, but just in case.  I assume
> this will blow up, but if it doesn't you can try running
> init-extent-tree again and see how that goes.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x557d47c7ebc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x557d47c7ebc0
Checking root 2 bytenr 15645018587136
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645018570752
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645019602944
Checking root 164629 bytenr 15645485137920
Checking root 164631 bytenr 15645496983552
Checking root 164633 bytenr 15645526884352
Checking root 164823 bytenr 15645999005696
Tree recovery finished, you can run check now

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x564088b52bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x564088b52bc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups
deleting space cache for 11106814787584
(...)
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
kernel-shared/ctree.c:1718: push_node_left: BUG_ON `check_sibling_keys(dst, src)` triggered, value 1
./btrfs(+0x1cf51)[0x56090b357f51]
./btrfs(+0x1ef96)[0x56090b359f96]
./btrfs(btrfs_search_slot+0xee4)[0x56090b35d881]
./btrfs(+0x8d592)[0x56090b3c8592]
./btrfs(+0x8d2c1)[0x56090b3c82c1]
./btrfs(+0x8d2c1)[0x56090b3c82c1]
./btrfs(+0x8d8b0)[0x56090b3c88b0]
./btrfs(+0x8d0bb)[0x56090b3c80bb]
./btrfs(btrfs_init_extent_tree+0xc83)[0x56090b3c9e48]
./btrfs(+0x8467e)[0x56090b3bf67e]
./btrfs(handle_command_group+0x49)[0x56090b35317b]
./btrfs(main+0x94)[0x56090b353275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f37ada9c7fd]
./btrfs(_start+0x2a)[0x56090b352e1a]
Aborted

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07  2:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-06-07  2:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-06-07  3:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-07  2:57 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Jun 6, 2022 at 10:37 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Jun 06, 2022 at 10:28:25PM -0400, Josef Bacik wrote:
> > Ok I thought I caught this particular problem but I don't, so I fixed
> > tree-recover to handle unordered keys in different nodes.  Pull and
> > build, run tree-recover.  It's going to start deleting slots for
> > unordered keys, picking whichever node is newer as the source of
> > truth.  You should only see this happen on 164624, if you see it fire
> > a bunch right away stop it and send me the output so I can make sure I
> > didn't screw anything up.  I went over the code and diff a few times
> > to make sure I didn't mess anything up, but I could have missed
> > something.  If that runs and fixes stuff, run it again just to make
> > sure it doesn't find anything the second time.  It shouldn't since I
> > re-start the loop if we adjust things, but just in case.  I assume
> > this will blow up, but if it doesn't you can try running
> > init-extent-tree again and see how that goes.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
> FS_INFO IS 0x557d47c7ebc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x557d47c7ebc0
> Checking root 2 bytenr 15645018587136
> Checking root 4 bytenr 15645019078656
> Checking root 5 bytenr 15645018161152
> Checking root 7 bytenr 15645018570752
> Checking root 9 bytenr 15645740367872
> Checking root 161197 bytenr 15645018341376
> Checking root 161199 bytenr 15645018652672
> Checking root 161200 bytenr 15645018750976
> Checking root 161889 bytenr 11160502124544
> Checking root 162628 bytenr 15645018931200
> Checking root 162632 bytenr 15645018210304
> Checking root 163298 bytenr 15645019045888
> Checking root 163302 bytenr 15645018685440
> Checking root 163303 bytenr 15645019095040
> Checking root 163316 bytenr 15645018996736
> Checking root 163920 bytenr 15645019144192
> Checking root 164620 bytenr 15645019275264
> Checking root 164623 bytenr 15645019226112
> Checking root 164624 bytenr 15645019602944
> Checking root 164629 bytenr 15645485137920
> Checking root 164631 bytenr 15645496983552
> Checking root 164633 bytenr 15645526884352
> Checking root 164823 bytenr 15645999005696
> Tree recovery finished, you can run check now
>

Ah my bad, added all the repair code but not the code to notice it was
broken, try again please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07  2:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-06-07  3:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-06-07 14:51                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-07  3:22 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Jun 06, 2022 at 10:57:17PM -0400, Josef Bacik wrote:
> Ah my bad, added all the repair code but not the code to notice it was
> broken, try again please.  Thanks,

FS_INFO IS 0x5641505c3bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5641505c3bc0
Checking root 2 bytenr 15645019553792
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645019521024
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645019602944
scanning, best has 0 found 0 bad
checking block 15645019602944 generation 2588164 fs info generation 2588170
trying bytenr 15645019602944 got 145 blocks 1 bad
checking block 15645502210048 generation 1601068 fs info generation 2588170
trying bytenr 15645502210048 got 146 blocks 1 bad
checking block 15645022208000 generation 1739020 fs info generation 2588170
scan for best root 164624 wants to use 15645502210048 as the root bytenr
Repairing root 164624 bad_blocks 1 update 1
Segmentation fault


again, under gdb
scanning, best has 0 found 0 bad
checking block 15645019602944 generation 2588164 fs info generation 2588170
trying bytenr 15645019602944 got 145 blocks 1 bad
checking block 15645502210048 generation 1601068 fs info generation 2588170
trying bytenr 15645502210048 got 122 blocks 0 bad
checking block 15645022208000 generation 1739020 fs info generation 2588170
scan for best root 164624 wants to use 15645019602944 as the root bytenr
Repairing root 164624 bad_blocks 1 update 1
we're pointing at an empty node, delete slot 1 in block 15645019602944

Program received signal SIGSEGV, Segmentation fault.
repair_tree (fs_info=fs_info@entry=0x555555651bc0, root_info=root_info@entry=0x7fffffffdc20, eb=eb@entry=0x55555574bfb0)
    at ./kernel-shared/ctree.h:2136
2136    BTRFS_SETGET_HEADER_FUNCS(header_generation, struct btrfs_header,

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
u

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07  3:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-06-07 14:51                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-07 14:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-07 15:18                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 2 replies; 479+ messages in thread
From: Josef Bacik @ 2022-06-07 14:51 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Jun 6, 2022 at 11:22 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Jun 06, 2022 at 10:57:17PM -0400, Josef Bacik wrote:
> > Ah my bad, added all the repair code but not the code to notice it was
> > broken, try again please.  Thanks,
>
> FS_INFO IS 0x5641505c3bc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x5641505c3bc0
> Checking root 2 bytenr 15645019553792
> Checking root 4 bytenr 15645019078656
> Checking root 5 bytenr 15645018161152
> Checking root 7 bytenr 15645019521024
> Checking root 9 bytenr 15645740367872
> Checking root 161197 bytenr 15645018341376
> Checking root 161199 bytenr 15645018652672
> Checking root 161200 bytenr 15645018750976
> Checking root 161889 bytenr 11160502124544
> Checking root 162628 bytenr 15645018931200
> Checking root 162632 bytenr 15645018210304
> Checking root 163298 bytenr 15645019045888
> Checking root 163302 bytenr 15645018685440
> Checking root 163303 bytenr 15645019095040
> Checking root 163316 bytenr 15645018996736
> Checking root 163920 bytenr 15645019144192
> Checking root 164620 bytenr 15645019275264
> Checking root 164623 bytenr 15645019226112
> Checking root 164624 bytenr 15645019602944
> scanning, best has 0 found 0 bad
> checking block 15645019602944 generation 2588164 fs info generation 2588170
> trying bytenr 15645019602944 got 145 blocks 1 bad
> checking block 15645502210048 generation 1601068 fs info generation 2588170
> trying bytenr 15645502210048 got 146 blocks 1 bad
> checking block 15645022208000 generation 1739020 fs info generation 2588170
> scan for best root 164624 wants to use 15645502210048 as the root bytenr
> Repairing root 164624 bad_blocks 1 update 1
> Segmentation fault
>
>
> again, under gdb
> scanning, best has 0 found 0 bad
> checking block 15645019602944 generation 2588164 fs info generation 2588170
> trying bytenr 15645019602944 got 145 blocks 1 bad
> checking block 15645502210048 generation 1601068 fs info generation 2588170
> trying bytenr 15645502210048 got 122 blocks 0 bad
> checking block 15645022208000 generation 1739020 fs info generation 2588170
> scan for best root 164624 wants to use 15645019602944 as the root bytenr
> Repairing root 164624 bad_blocks 1 update 1
> we're pointing at an empty node, delete slot 1 in block 15645019602944
>
> Program received signal SIGSEGV, Segmentation fault.
> repair_tree (fs_info=fs_info@entry=0x555555651bc0, root_info=root_info@entry=0x7fffffffdc20, eb=eb@entry=0x55555574bfb0)
>     at ./kernel-shared/ctree.h:2136
> 2136    BTRFS_SETGET_HEADER_FUNCS(header_generation, struct btrfs_header,

Hmm weird, I think I spotted it, give it a try again please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 14:51                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-06-07 14:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-07 15:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-06-07 15:18                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-07 14:53 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Jun 07, 2022 at 10:51:35AM -0400, Josef Bacik wrote:
> Hmm weird, I think I spotted it, give it a try again please.  Thanks,

scanning, best has 0 found 0 bad
checking block 15645485137920 generation 1601075 fs info generation 2588170
trying bytenr 15645485137920 got 99 blocks 1 bad
scan for best root 164629 wants to use 15645485137920 as the root bytenr
Repairing root 164629 bad_blocks 1 update 1
we're pointing at an empty node, delete slot 2 in block 15645485137920
kernel-shared/extent_io.c:664: free_extent_buffer_internal: BUG_ON `eb->refs < 0` triggered, value 1
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x311a0)[0x5555555851a0]
/var/local/src/btrfs-progs-josefbacik/btrfs(free_extent_buffer_nocache+0xe)[0x555555585a63]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bb02)[0x5555555dfb02]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8c1ac)[0x5555555e01ac]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_recover_trees+0x446)[0x5555555e0c4d]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8475a)[0x5555555d875a]
/var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b]
/var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd]
/var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 14:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-07 15:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 0 replies; 479+ messages in thread
From: Josef Bacik @ 2022-06-07 15:00 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Jun 7, 2022 at 10:53 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Jun 07, 2022 at 10:51:35AM -0400, Josef Bacik wrote:
> > Hmm weird, I think I spotted it, give it a try again please.  Thanks,
>
> scanning, best has 0 found 0 bad
> checking block 15645485137920 generation 1601075 fs info generation 2588170
> trying bytenr 15645485137920 got 99 blocks 1 bad
> scan for best root 164629 wants to use 15645485137920 as the root bytenr
> Repairing root 164629 bad_blocks 1 update 1
> we're pointing at an empty node, delete slot 2 in block 15645485137920
> kernel-shared/extent_io.c:664: free_extent_buffer_internal: BUG_ON `eb->refs < 0` triggered, value 1
> /var/local/src/btrfs-progs-josefbacik/btrfs(+0x311a0)[0x5555555851a0]
> /var/local/src/btrfs-progs-josefbacik/btrfs(free_extent_buffer_nocache+0xe)[0x555555585a63]
> /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bb02)[0x5555555dfb02]
> /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8c1ac)[0x5555555e01ac]
> /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_recover_trees+0x446)[0x5555555e0c4d]
> /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8475a)[0x5555555d875a]
> /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b]
> /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd]
> /var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]
>

Sigh I had that thought and then didn't type it.  Try again please, thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 14:51                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-07 14:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-07 15:18                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-07 15:21                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-07 15:18 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Jun 07, 2022 at 10:51:35AM -0400, Josef Bacik wrote:
> Hmm weird, I think I spotted it, give it a try again please.  Thanks,

No more errors, and now init-extent-tree is deleting a lot of inodes again, let it continue?

FS_INFO IS 0x555555651bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555651bc0
Checking root 2 bytenr 15645019553792
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645019521024
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645019602944
Checking root 164629 bytenr 15645485137920
Checking root 164631 bytenr 15645496983552
Checking root 164633 bytenr 15645526884352
Checking root 164823 bytenr 15645999005696
Tree recovery finished, you can run check now

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x56090c17bbc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x56090c17bbc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups
deleting space cache for 11106814787584
deleting space cache for 11108962271232
deleting space cache for 11110036013056
deleting space cache for 11111109754880
(...)
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
push node left from right mid 15645019717632 nritems 40 right 11160502419456 nritems 0 parent 15645019684864 parent nritems 5
parent nritems is now 4

(...)

searching 164624 for bad extents
processed 16384 of 109264896 possible bytes, 0%
Found an extent we don't have a block group for in the file
ref to path failed
Couldn't find any paths for this inode
Deleting [790, 108, 268435456] root 15645019471872 path top 15645019471872 top slot 1 leaf 15645019684864 slot 15



-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 15:18                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-07 15:21                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-06-07 15:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-07 15:21 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Jun 7, 2022 at 11:18 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Jun 07, 2022 at 10:51:35AM -0400, Josef Bacik wrote:
> > Hmm weird, I think I spotted it, give it a try again please.  Thanks,
>
> No more errors, and now init-extent-tree is deleting a lot of inodes again, let it continue?
>
> FS_INFO IS 0x555555651bc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x555555651bc0
> Checking root 2 bytenr 15645019553792
> Checking root 4 bytenr 15645019078656
> Checking root 5 bytenr 15645018161152
> Checking root 7 bytenr 15645019521024
> Checking root 9 bytenr 15645740367872
> Checking root 161197 bytenr 15645018341376
> Checking root 161199 bytenr 15645018652672
> Checking root 161200 bytenr 15645018750976
> Checking root 161889 bytenr 11160502124544
> Checking root 162628 bytenr 15645018931200
> Checking root 162632 bytenr 15645018210304
> Checking root 163298 bytenr 15645019045888
> Checking root 163302 bytenr 15645018685440
> Checking root 163303 bytenr 15645019095040
> Checking root 163316 bytenr 15645018996736
> Checking root 163920 bytenr 15645019144192
> Checking root 164620 bytenr 15645019275264
> Checking root 164623 bytenr 15645019226112
> Checking root 164624 bytenr 15645019602944
> Checking root 164629 bytenr 15645485137920
> Checking root 164631 bytenr 15645496983552
> Checking root 164633 bytenr 15645526884352
> Checking root 164823 bytenr 15645999005696
> Tree recovery finished, you can run check now
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
> FS_INFO IS 0x56090c17bbc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x56090c17bbc0
> Walking all our trees and pinning down the currently accessible blocks
> Clearing the extent root and re-init'ing the block groups
> deleting space cache for 11106814787584
> deleting space cache for 11108962271232
> deleting space cache for 11110036013056
> deleting space cache for 11111109754880
> (...)
> inserting block group 15932780969984
> inserting block group 15933854711808

Can you capture all of these lines and paste them?  We found a bunch
of old block groups but we may not have found everything.  I might
want to try manually going and looking for those chunks just so we can
avoid mass deleting things.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 15:21                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-06-07 15:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-06-07 17:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-07 15:32 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Jun 07, 2022 at 11:21:57AM -0400, Josef Bacik wrote:
> Can you capture all of these lines and paste them?  We found a bunch
> of old block groups but we may not have found everything.  I might
> want to try manually going and looking for those chunks just so we can
> avoid mass deleting things.  Thanks,

https://pastebin.com/dPzJgVU9
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 15:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-06-07 17:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-06-07 18:27                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-07 17:56 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Jun 7, 2022 at 11:32 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Jun 07, 2022 at 11:21:57AM -0400, Josef Bacik wrote:
> > Can you capture all of these lines and paste them?  We found a bunch
> > of old block groups but we may not have found everything.  I might
> > want to try manually going and looking for those chunks just so we can
> > avoid mass deleting things.  Thanks,
>
> https://pastebin.com/dPzJgVU9

Ok re-run it, it'll crash right as it tries to delete something, I
need the bytenr it's complaining about.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 17:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-06-07 18:27                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-06-07 19:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-07 18:27 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Jun 07, 2022 at 01:56:38PM -0400, Josef Bacik wrote:
> On Tue, Jun 7, 2022 at 11:32 AM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Tue, Jun 07, 2022 at 11:21:57AM -0400, Josef Bacik wrote:
> > > Can you capture all of these lines and paste them?  We found a bunch
> > > of old block groups but we may not have found everything.  I might
> > > want to try manually going and looking for those chunks just so we can
> > > avoid mass deleting things.  Thanks,
> >
> > https://pastebin.com/dPzJgVU9
> 
> Ok re-run it, it'll crash right as it tries to delete something, I
> need the bytenr it's complaining about.  Thanks,

inserting block group 15929559744512
inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
processed 49152 of 109264896 possible bytes, 0%
Found an extent we don't have a block group for in the file 1258276585472
cmds/rescue-init-extent-tree.c:246: process_leaf_item: BUG_ON `bytenr` triggered, value 1258276585472
./btrfs(+0x8cef8)[0x5596c34b4ef8]
./btrfs(+0x8d685)[0x5596c34b5685]
./btrfs(+0x8d47c)[0x5596c34b547c]
./btrfs(+0x8da64)[0x5596c34b5a64]
./btrfs(+0x8d28c)[0x5596c34b528c]
./btrfs(btrfs_init_extent_tree+0xc83)[0x5596c34b6ffc]
./btrfs(+0x8467e)[0x5596c34ac67e]
./btrfs(handle_command_group+0x49)[0x5596c344017b]
./btrfs(main+0x94)[0x5596c3440275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f4ff8fc97fd]
./btrfs(_start+0x2a)[0x5596c343fe1a]
Aborted

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 18:27                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-06-07 19:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-06-07 19:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-07 19:46 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Jun 7, 2022 at 2:27 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Jun 07, 2022 at 01:56:38PM -0400, Josef Bacik wrote:
> > On Tue, Jun 7, 2022 at 11:32 AM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Tue, Jun 07, 2022 at 11:21:57AM -0400, Josef Bacik wrote:
> > > > Can you capture all of these lines and paste them?  We found a bunch
> > > > of old block groups but we may not have found everything.  I might
> > > > want to try manually going and looking for those chunks just so we can
> > > > avoid mass deleting things.  Thanks,
> > >
> > > https://pastebin.com/dPzJgVU9
> >
> > Ok re-run it, it'll crash right as it tries to delete something, I
> > need the bytenr it's complaining about.  Thanks,
>
> inserting block group 15929559744512
> inserting block group 15930633486336
> inserting block group 15931707228160
> inserting block group 15932780969984
> inserting block group 15933854711808
> ERROR: Error reading data reloc tree -2
>
> ERROR: failed to reinit the data reloc root
> searching 1 for bad extents
> processed 999424 of 0 possible bytes, 0%
> searching 4 for bad extents
> processed 163840 of 1064960 possible bytes, 15%
> searching 5 for bad extents
> processed 65536 of 10960896 possible bytes, 0%
> searching 7 for bad extents
> processed 16384 of 16570974208 possible bytes, 0%
> searching 9 for bad extents
> processed 16384 of 16384 possible bytes, 100%
> searching 161197 for bad extents
> processed 131072 of 108986368 possible bytes, 0%
> searching 161199 for bad extents
> processed 196608 of 49479680 possible bytes, 0%
> searching 161200 for bad extents
> processed 180224 of 254214144 possible bytes, 0%
> searching 161889 for bad extents
> processed 229376 of 49446912 possible bytes, 0%
> searching 162628 for bad extents
> processed 49152 of 49463296 possible bytes, 0%
> searching 162632 for bad extents
> processed 147456 of 94633984 possible bytes, 0%
> searching 163298 for bad extents
> processed 49152 of 49463296 possible bytes, 0%
> searching 163302 for bad extents
> processed 147456 of 94633984 possible bytes, 0%
> searching 163303 for bad extents
> processed 131072 of 76333056 possible bytes, 0%
> searching 163316 for bad extents
> processed 147456 of 108544000 possible bytes, 0%
> searching 163920 for bad extents
> processed 16384 of 108691456 possible bytes, 0%
> searching 164620 for bad extents
> processed 49152 of 49463296 possible bytes, 0%
> searching 164623 for bad extents
> processed 311296 of 63193088 possible bytes, 0%
> searching 164624 for bad extents
> processed 49152 of 109264896 possible bytes, 0%
> Found an extent we don't have a block group for in the file 1258276585472
> cmds/rescue-init-extent-tree.c:246: process_leaf_item: BUG_ON `bytenr` triggered, value 1258276585472

Perfect, this isn't in our list, so we definitely don't have it.  I've
added some debugging to recover-chunks, can you run btrfs rescue
recover-chunks and capture it's output?  We may not find this chunk
and in that case it needs to just delete stuff, but I'd like to try a
little harder before we give up.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 19:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-06-07 19:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-06-07 20:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-07 19:57 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Jun 07, 2022 at 03:46:12PM -0400, Josef Bacik wrote:
> 
> Perfect, this isn't in our list, so we definitely don't have it.  I've
> added some debugging to recover-chunks, can you run btrfs rescue
> recover-chunks and capture it's output?  We may not find this chunk
> and in that case it needs to just delete stuff, but I'd like to try a
> little harder before we give up.  Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
FS_INFO IS 0x560392adcbc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x560392adcbc0
Walking all our trees and pinning down the currently accessible blocks
No missing chunks, we're all done
doing close???
Recover chunks succeeded, you can run check now

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x5621bb7a7bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5621bb7a7bc0
Checking root 2 bytenr 15645019504640
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645019439104
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645018226688
Checking root 164629 bytenr 15645485137920
Checking root 164631 bytenr 15645496983552
Checking root 164633 bytenr 15645526884352
Checking root 164823 bytenr 15645999005696
Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# 

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 19:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-06-07 20:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-07 20:25                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-07 20:44                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 2 replies; 479+ messages in thread
From: Josef Bacik @ 2022-06-07 20:10 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Jun 7, 2022 at 3:57 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Jun 07, 2022 at 03:46:12PM -0400, Josef Bacik wrote:
> >
> > Perfect, this isn't in our list, so we definitely don't have it.  I've
> > added some debugging to recover-chunks, can you run btrfs rescue
> > recover-chunks and capture it's output?  We may not find this chunk
> > and in that case it needs to just delete stuff, but I'd like to try a
> > little harder before we give up.  Thanks,
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> FS_INFO IS 0x560392adcbc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x560392adcbc0
> Walking all our trees and pinning down the currently accessible blocks
> No missing chunks, we're all done
> doing close???
> Recover chunks succeeded, you can run check now

Alright it looks like we've gotten everything we're going to get, go
ahead and let init-extent-tree do its thing.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 20:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-06-07 20:25                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-07 20:44                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  1 sibling, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-06-07 20:25 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Jun 07, 2022 at 04:10:02PM -0400, Josef Bacik wrote:
> Alright it looks like we've gotten everything we're going to get, go
> ahead and let init-extent-tree do its thing.  Thanks,

Thanks.

Currently I see
searching 164624 for bad extents
processed 655360 of 109199360 possible bytes, 0%
Found an extent we don't have a block group for in the file 6708223688704
ref to path failed
Couldn't find any paths for this inode
Deleting [25834, 108, 15728640] root 15645019504640 path top 15645019504640 top slot 40 leaf 15645020176384 slot 2

Mostly those above, with a few of these:
searching 164624 for bad extents
processed 49152 of 109264896 possible bytes, 0%
Found an extent we don't have a block group for in the file 1259115134976
file
Deleting [796, 108, 454770688] root 15645018243072 path top 15645018243072 top slot 3 leaf 15645019176960 slot 43

Thanks for the progress report (0% here), I expect it will take many days.
Will report back

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 20:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-07 20:25                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-07 20:44                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-07 20:58                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-07 20:44 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Jun 07, 2022 at 04:10:02PM -0400, Josef Bacik wrote:
> Alright it looks like we've gotten everything we're going to get, go
> ahead and let init-extent-tree do its thing.  Thanks,

Argh, it ran for a while and then it crashed, but I didn't have it run under gdb.

searching 164624 for bad extents
processed 655360 of 109084672 possible bytes, 0%
Found an extent we don't have a block group for in the file 12283412910080
ref to path failed
Couldn't find any paths for this inode
Deleting [26761, 108, 34557952] root 15645018243072 path top 15645018243072 top slot 40 leaf 15645020192768 slot 3

searching 164624 for bad extents
processed 655360 of 109084672 possible bytes, 0%
Found an extent we don't have a block group for in the file 12283452309504
ref to path failed
Couldn't find any paths for this inode
Deleting [26761, 108, 43954176] root 15645019504640 path top 15645019504640 top slot 40 leaf 15645020028928 slot 3
Segmentation fault

re-running it thankfully crashed in the same place
processed 655360 of 109084672 possible bytes, 0%
Found an extent we don't have a block group for in the file 12283452309504
ref to path failed
Couldn't find any paths for this inode
Deleting [26761, 108, 43954176] root 15645019586560 path top 15645019586560 top slot 40 leaf 15645019602944 slot 3

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff78e1c8f in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff78e1c8f in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00005555555862f9 in memcpy (__len=17, __src=<optimized out>, __dest=0x7fffffffd5d0)
    at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:29
#2  read_extent_buffer (eb=<optimized out>, dst=dst@entry=0x7fffffffd630, start=<optimized out>, len=len@entry=17)
    at kernel-shared/extent_io.c:1002
#3  0x00005555555735aa in btrfs_node_key (nr=<optimized out>, disk_key=0x7fffffffd630, eb=<optimized out>)
    at ./kernel-shared/ctree.h:2113
#4  btrfs_node_key_to_cpu (nr=<optimized out>, key=0x7fffffffd650, eb=<optimized out>) at ./kernel-shared/ctree.h:2113
#5  check_block (fs_info=0x555555651bc0, path=path@entry=0x7fffffffd760, level=level@entry=49) at kernel-shared/ctree.c:854
#6  0x0000555555577ecb in btrfs_del_items (trans=trans@entry=0x555555838ff0, root=root@entry=0x55555582c7b0, 
    path=path@entry=0x7fffffffd760, slot=40, nr=nr@entry=1) at kernel-shared/ctree.c:3163
#7  0x00005555555e18d1 in btrfs_del_item (path=0x7fffffffd760, root=0x55555582c7b0, trans=0x555555838ff0)
    at ./kernel-shared/ctree.h:2781
#8  delete_item (key=0x7fffffffd740, root=0x55555582c7b0) at cmds/rescue-init-extent-tree.c:208
#9  process_leaf_item (slot=<optimized out>, eb=0x0, root=0x55555582c7b0) at cmds/rescue-init-extent-tree.c:297
#10 look_for_bad_extents (root=root@entry=0x55555582c7b0, eb=eb@entry=0x555555834d90, current=current@entry=0x7fffffffda18)
    at cmds/rescue-init-extent-tree.c:338
#11 0x00005555555e1468 in look_for_bad_extents (root=root@entry=0x55555582c7b0, eb=0x5555557a4bb0, 
    current=current@entry=0x7fffffffda18) at cmds/rescue-init-extent-tree.c:331
#12 0x00005555555e1a66 in clear_bad_extents (root=0x55555582c7b0) at cmds/rescue-init-extent-tree.c:362
#13 0x00005555555e1262 in foreach_root (fs_info=fs_info@entry=0x555555651bc0, cb=cb@entry=0x5555555e1a21 <clear_bad_extents>)
    at cmds/rescue-init-extent-tree.c:152
#14 0x00005555555e2ffe in reinit_extent_tree (fs_info=0x555555651bc0) at cmds/rescue-init-extent-tree.c:873
#15 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:1219
#16 0x00005555555d867e in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:139
#17 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555647c80 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#18 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#19 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555648cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#20 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 20:44                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-07 20:58                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-06-07 21:25                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-07 20:58 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Jun 7, 2022 at 4:44 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Jun 07, 2022 at 04:10:02PM -0400, Josef Bacik wrote:
> > Alright it looks like we've gotten everything we're going to get, go
> > ahead and let init-extent-tree do its thing.  Thanks,
>
> Argh, it ran for a while and then it crashed, but I didn't have it run under gdb.
>
> searching 164624 for bad extents
> processed 655360 of 109084672 possible bytes, 0%
> Found an extent we don't have a block group for in the file 12283412910080
> ref to path failed
> Couldn't find any paths for this inode
> Deleting [26761, 108, 34557952] root 15645018243072 path top 15645018243072 top slot 40 leaf 15645020192768 slot 3
>
> searching 164624 for bad extents
> processed 655360 of 109084672 possible bytes, 0%
> Found an extent we don't have a block group for in the file 12283452309504
> ref to path failed
> Couldn't find any paths for this inode
> Deleting [26761, 108, 43954176] root 15645019504640 path top 15645019504640 top slot 40 leaf 15645020028928 slot 3
> Segmentation fault
>
> re-running it thankfully crashed in the same place
> processed 655360 of 109084672 possible bytes, 0%
> Found an extent we don't have a block group for in the file 12283452309504
> ref to path failed
> Couldn't find any paths for this inode
> Deleting [26761, 108, 43954176] root 15645019586560 path top 15645019586560 top slot 40 leaf 15645019602944 slot 3
>

Ok I think I fixed that, but I also updated the loop to bulk delete as
many bad items in a leaf at a time, hopefully this will make it go
much faster.  Expect more fireworks with the new code.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 20:58                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-06-07 21:25                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-06-07 23:33                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-07 21:25 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Jun 07, 2022 at 04:58:57PM -0400, Josef Bacik wrote:
> Ok I think I fixed that, but I also updated the loop to bulk delete as
> many bad items in a leaf at a time, hopefully this will make it go
> much faster.  Expect more fireworks with the new code.  Thanks,

Found an extent we don't have a block group for in the file 10741731311616
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 0] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020766208 slot 26 nr 73

searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 10741731311616

Found an extent we don't have a block group for in the file 743378268160

Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 427687936] root 15645019537408 path top 15645019537408 top slot 49 leaf 15645020766208 slot 26 nr 148

searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 1747189760] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020782592 slot 26 nr 1

searching 164629 for bad extents
processed 835584 of 108838912 possible bytes, 0%
corrupt leaf: root=164629 root bytenr 15645020241920 commit bytenr 0 block=15645020438528 physical=15054974140416 slot=0 level 0, invalid level for node, have 0 expect [1, 7]
kernel-shared/ctree.c:148: btrfs_release_path: BUG_ON `ret` triggered, value -5
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_release_path+0x62)[0x555555573667]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8d1fe)[0x5555555e11fe]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_init_extent_tree+0xc83)[0x5555555e2fd0]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x845dd)[0x5555555d85dd]
/var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b]
/var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd]
/var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]

Program received signal SIGABRT, Aborted.
0x00007ffff78768a1 in raise () from /lib/x86_64-linux-gnu/libc.so.6

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 21:25                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-06-07 23:33                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-06-07 23:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-07 23:33 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Jun 7, 2022 at 5:25 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Jun 07, 2022 at 04:58:57PM -0400, Josef Bacik wrote:
> > Ok I think I fixed that, but I also updated the loop to bulk delete as
> > many bad items in a leaf at a time, hopefully this will make it go
> > much faster.  Expect more fireworks with the new code.  Thanks,
>
> Found an extent we don't have a block group for in the file 10741731311616
> ref to path failed
> Couldn't find any paths for this inode
> Deleting [73101, 108, 0] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020766208 slot 26 nr 73
>
> searching 164629 for bad extents
> processed 802816 of 108838912 possible bytes, 0%
> Found an extent we don't have a block group for in the file 10741731311616
>
> Found an extent we don't have a block group for in the file 743378268160
>
> Found an extent we don't have a block group for in the file 946736541696
> ref to path failed
> Couldn't find any paths for this inode
> Deleting [73101, 108, 427687936] root 15645019537408 path top 15645019537408 top slot 49 leaf 15645020766208 slot 26 nr 148
>
> searching 164629 for bad extents
> processed 802816 of 108838912 possible bytes, 0%
> Found an extent we don't have a block group for in the file 946736541696
> ref to path failed
> Couldn't find any paths for this inode
> Deleting [73101, 108, 1747189760] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020782592 slot 26 nr 1
>
> searching 164629 for bad extents
> processed 835584 of 108838912 possible bytes, 0%
> corrupt leaf: root=164629 root bytenr 15645020241920 commit bytenr 0 block=15645020438528 physical=15054974140416 slot=0 level 0, invalid level for node, have 0 expect [1, 7]
> kernel-shared/ctree.c:148: btrfs_release_path: BUG_ON `ret` triggered, value -5

Ugh come on, this must get triggered because we clean up some stuff
and then the corrupt blocks are suddenly right next to eachother.  Can
you re-run tree-recover and see if it deletes those keys?  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 23:33                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-06-07 23:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-06-07 23:41                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-07 23:37 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Jun 07, 2022 at 07:33:00PM -0400, Josef Bacik wrote:
> On Tue, Jun 7, 2022 at 5:25 PM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Tue, Jun 07, 2022 at 04:58:57PM -0400, Josef Bacik wrote:
> > > Ok I think I fixed that, but I also updated the loop to bulk delete as
> > > many bad items in a leaf at a time, hopefully this will make it go
> > > much faster.  Expect more fireworks with the new code.  Thanks,
> >
> > Found an extent we don't have a block group for in the file 10741731311616
> > ref to path failed
> > Couldn't find any paths for this inode
> > Deleting [73101, 108, 0] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020766208 slot 26 nr 73
> >
> > searching 164629 for bad extents
> > processed 802816 of 108838912 possible bytes, 0%
> > Found an extent we don't have a block group for in the file 10741731311616
> >
> > Found an extent we don't have a block group for in the file 743378268160
> >
> > Found an extent we don't have a block group for in the file 946736541696
> > ref to path failed
> > Couldn't find any paths for this inode
> > Deleting [73101, 108, 427687936] root 15645019537408 path top 15645019537408 top slot 49 leaf 15645020766208 slot 26 nr 148
> >
> > searching 164629 for bad extents
> > processed 802816 of 108838912 possible bytes, 0%
> > Found an extent we don't have a block group for in the file 946736541696
> > ref to path failed
> > Couldn't find any paths for this inode
> > Deleting [73101, 108, 1747189760] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020782592 slot 26 nr 1
> >
> > searching 164629 for bad extents
> > processed 835584 of 108838912 possible bytes, 0%
> > corrupt leaf: root=164629 root bytenr 15645020241920 commit bytenr 0 block=15645020438528 physical=15054974140416 slot=0 level 0, invalid level for node, have 0 expect [1, 7]
> > kernel-shared/ctree.c:148: btrfs_release_path: BUG_ON `ret` triggered, value -5
> 
> Ugh come on, this must get triggered because we clean up some stuff
> and then the corrupt blocks are suddenly right next to eachother.  Can
> you re-run tree-recover and see if it deletes those keys?  Thanks,

Sorry :(

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x559148b31bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x559148b31bc0
Checking root 2 bytenr 15645020913664
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645019586560
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645018275840
Checking root 164629 bytenr 15645020241920
Checking root 164631 bytenr 15645496983552
Checking root 164633 bytenr 15645526884352
Checking root 164823 bytenr 15645999005696
Tree recovery finished, you can run check now
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 23:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-06-07 23:41                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-06-08  0:07                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-07 23:41 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Jun 7, 2022 at 7:37 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Jun 07, 2022 at 07:33:00PM -0400, Josef Bacik wrote:
> > On Tue, Jun 7, 2022 at 5:25 PM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Tue, Jun 07, 2022 at 04:58:57PM -0400, Josef Bacik wrote:
> > > > Ok I think I fixed that, but I also updated the loop to bulk delete as
> > > > many bad items in a leaf at a time, hopefully this will make it go
> > > > much faster.  Expect more fireworks with the new code.  Thanks,
> > >
> > > Found an extent we don't have a block group for in the file 10741731311616
> > > ref to path failed
> > > Couldn't find any paths for this inode
> > > Deleting [73101, 108, 0] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020766208 slot 26 nr 73
> > >
> > > searching 164629 for bad extents
> > > processed 802816 of 108838912 possible bytes, 0%
> > > Found an extent we don't have a block group for in the file 10741731311616
> > >
> > > Found an extent we don't have a block group for in the file 743378268160
> > >
> > > Found an extent we don't have a block group for in the file 946736541696
> > > ref to path failed
> > > Couldn't find any paths for this inode
> > > Deleting [73101, 108, 427687936] root 15645019537408 path top 15645019537408 top slot 49 leaf 15645020766208 slot 26 nr 148
> > >
> > > searching 164629 for bad extents
> > > processed 802816 of 108838912 possible bytes, 0%
> > > Found an extent we don't have a block group for in the file 946736541696
> > > ref to path failed
> > > Couldn't find any paths for this inode
> > > Deleting [73101, 108, 1747189760] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020782592 slot 26 nr 1
> > >
> > > searching 164629 for bad extents
> > > processed 835584 of 108838912 possible bytes, 0%
> > > corrupt leaf: root=164629 root bytenr 15645020241920 commit bytenr 0 block=15645020438528 physical=15054974140416 slot=0 level 0, invalid level for node, have 0 expect [1, 7]
> > > kernel-shared/ctree.c:148: btrfs_release_path: BUG_ON `ret` triggered, value -5
> >
> > Ugh come on, this must get triggered because we clean up some stuff
> > and then the corrupt blocks are suddenly right next to eachother.  Can
> > you re-run tree-recover and see if it deletes those keys?  Thanks,
>
> Sorry :(
>

Ok I think it's just the check is wrong, I removed the check, maybe
I'll get lucky for once?  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-07 23:41                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-06-08  0:07                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-06-08  0:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-08  0:07 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Jun 07, 2022 at 07:41:32PM -0400, Josef Bacik wrote:
> > > Ugh come on, this must get triggered because we clean up some stuff
> > > and then the corrupt blocks are suddenly right next to eachother.  Can
> > > you re-run tree-recover and see if it deletes those keys?  Thanks,
> >
> > Sorry :(
> >
> Ok I think it's just the check is wrong, I removed the check, maybe
> I'll get lucky for once?  Thanks,

that helped.

(...)
Found an extent we don't have a block group for in the file 652682866688

Found an extent we don't have a block group for in the file 652682862592

Found an extent we don't have a block group for in the file 652723212288

Found an extent we don't have a block group for in the file 390870233088
ref to path failed
Couldn't find any paths for this inode
Deleting [69105, 108, 0] root 15645018226688 path top 15645018226688 top slot 11 leaf 15645020864512 slot 1 nr 118

searching 164823 for bad extents
processed 163840 of 63193088 possible bytes, 0%
Found an extent we don't have a block group for in the file 390870233088
Couldn't find any paths for this inode
Deleting [69134, 108, 36012032] root 15645018587136 path top 15645018587136 top slot 10 leaf 15645019553792 slot 45 nr 1

searching 164823 for bad extents
processed 196608 of 63193088 possible bytes, 0%
Recording extents for root 3
processed 180224 of 0 possible bytes, 0%
Recording extents for root 1
processed 999424 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 163840 of 1064960 possible bytes, 15%
Recording extents for root 5
processed 65536 of 10960896 possible bytes, 0%
Recording extents for root 7
processed 16384 of 16570974208 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 161197
processed 131072 of 108986368 possible bytes, 0%
Recording extents for root 161199
processed 196608 of 49479680 possible bytes, 0%
Recording extents for root 161200
processed 180224 of 254214144 possible bytes, 0%
Recording extents for root 161889
processed 196608 of 49446912 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 162632
processed 114688 of 94633984 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 163302
processed 98304 of 94633984 possible bytes, 0%
Recording extents for root 163303
processed 131072 of 76333056 possible bytes, 0%
Recording extents for root 163316
processed 98304 of 108544000 possible bytes, 0%
Recording extents for root 163920
processed 16384 of 108691456 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 164623
processed 311296 of 63193088 possible bytes, 0%
Recording extents for root 164624
processed 933888 of 108822528 possible bytes, 0%
Recording extents for root 164629
processed 622592 of 108838912 possible bytes, 0%
Recording extents for root 164631
ERROR: failed to insert the ref for the root block -17
wtf
it failed?? -17
doing close???
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=32768
extent buffer leak: start 15645019488256 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019488256 len 16384
extent buffer leak: start 15645020061696 len 16384
WARNING: dirty eb leak (aborted trans): start 15645020061696 len 16384
Init extent tree failed
[Inferior 1 (process 7259) exited with code 0357]

So can I go back to  check --repair /dev/mapper/dshelf1  ?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-08  0:07                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-06-08  0:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-08  0:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-08  0:32 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Jun 7, 2022 at 8:07 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Jun 07, 2022 at 07:41:32PM -0400, Josef Bacik wrote:
> > > > Ugh come on, this must get triggered because we clean up some stuff
> > > > and then the corrupt blocks are suddenly right next to eachother.  Can
> > > > you re-run tree-recover and see if it deletes those keys?  Thanks,
> > >
> > > Sorry :(
> > >
> > Ok I think it's just the check is wrong, I removed the check, maybe
> > I'll get lucky for once?  Thanks,
>
> that helped.
>
> (...)
> Found an extent we don't have a block group for in the file 652682866688
>
> Found an extent we don't have a block group for in the file 652682862592
>
> Found an extent we don't have a block group for in the file 652723212288
>
> Found an extent we don't have a block group for in the file 390870233088
> ref to path failed
> Couldn't find any paths for this inode
> Deleting [69105, 108, 0] root 15645018226688 path top 15645018226688 top slot 11 leaf 15645020864512 slot 1 nr 118
>
> searching 164823 for bad extents
> processed 163840 of 63193088 possible bytes, 0%
> Found an extent we don't have a block group for in the file 390870233088
> Couldn't find any paths for this inode
> Deleting [69134, 108, 36012032] root 15645018587136 path top 15645018587136 top slot 10 leaf 15645019553792 slot 45 nr 1
>
> searching 164823 for bad extents
> processed 196608 of 63193088 possible bytes, 0%
> Recording extents for root 3
> processed 180224 of 0 possible bytes, 0%
> Recording extents for root 1
> processed 999424 of 0 possible bytes, 0%
> doing roots
> Recording extents for root 4
> processed 163840 of 1064960 possible bytes, 15%
> Recording extents for root 5
> processed 65536 of 10960896 possible bytes, 0%
> Recording extents for root 7
> processed 16384 of 16570974208 possible bytes, 0%
> Recording extents for root 9
> processed 16384 of 16384 possible bytes, 100%
> Recording extents for root 161197
> processed 131072 of 108986368 possible bytes, 0%
> Recording extents for root 161199
> processed 196608 of 49479680 possible bytes, 0%
> Recording extents for root 161200
> processed 180224 of 254214144 possible bytes, 0%
> Recording extents for root 161889
> processed 196608 of 49446912 possible bytes, 0%
> Recording extents for root 162628
> processed 49152 of 49463296 possible bytes, 0%
> Recording extents for root 162632
> processed 114688 of 94633984 possible bytes, 0%
> Recording extents for root 163298
> processed 49152 of 49463296 possible bytes, 0%
> Recording extents for root 163302
> processed 98304 of 94633984 possible bytes, 0%
> Recording extents for root 163303
> processed 131072 of 76333056 possible bytes, 0%
> Recording extents for root 163316
> processed 98304 of 108544000 possible bytes, 0%
> Recording extents for root 163920
> processed 16384 of 108691456 possible bytes, 0%
> Recording extents for root 164620
> processed 49152 of 49463296 possible bytes, 0%
> Recording extents for root 164623
> processed 311296 of 63193088 possible bytes, 0%
> Recording extents for root 164624
> processed 933888 of 108822528 possible bytes, 0%
> Recording extents for root 164629
> processed 622592 of 108838912 possible bytes, 0%
> Recording extents for root 164631
> ERROR: failed to insert the ref for the root block -17
> wtf
> it failed?? -17

That shouldn't happen.  I reworked the code a bit and added some
debugging, re-run init-extent-tree please.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-08  0:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-06-08  0:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-08  1:31                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-08  0:42 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Jun 07, 2022 at 08:32:33PM -0400, Josef Bacik wrote:
> > processed 622592 of 108838912 possible bytes, 0%
> > Recording extents for root 164631
> > ERROR: failed to insert the ref for the root block -17
> > wtf
> > it failed?? -17
> 
> That shouldn't happen.  I reworked the code a bit and added some
> debugging, re-run init-extent-tree please.  Thanks,

processed 196608 of 63193088 possible bytes, 0%
searching 18446744073709551607 for bad extents
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 3
processed 180224 of 0 possible bytes, 0%
Recording extents for root 1
ERROR: commit_root already set when starting transaction
ERROR: error starting transaction
doing close???
extent buffer leak: start 11160502779904 len 16384
extent buffer leak: start 15645018308608 len 16384
Init extent tree failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-08  0:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-08  1:31                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-06-08  2:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-08  1:31 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Jun 7, 2022 at 8:42 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Jun 07, 2022 at 08:32:33PM -0400, Josef Bacik wrote:
> > > processed 622592 of 108838912 possible bytes, 0%
> > > Recording extents for root 164631
> > > ERROR: failed to insert the ref for the root block -17
> > > wtf
> > > it failed?? -17
> >
> > That shouldn't happen.  I reworked the code a bit and added some
> > debugging, re-run init-extent-tree please.  Thanks,
>
> processed 196608 of 63193088 possible bytes, 0%
> searching 18446744073709551607 for bad extents
> processed 16384 of 16384 possible bytes, 100%
> Recording extents for root 3
> processed 180224 of 0 possible bytes, 0%
> Recording extents for root 1
> ERROR: commit_root already set when starting transaction
> ERROR: error starting transaction
> doing close???
> extent buffer leak: start 11160502779904 len 16384
> extent buffer leak: start 15645018308608 len 16384
> Init extent tree failed

I swear first thing I'm doing after all this is updating the shared
code.  Try again please,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-08  1:31                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-06-08  2:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-06-08 20:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-08  2:12 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Tue, Jun 07, 2022 at 09:31:28PM -0400, Josef Bacik wrote:
> > doing close???
> > extent buffer leak: start 11160502779904 len 16384
> > extent buffer leak: start 15645018308608 len 16384
> > Init extent tree failed
> 
> I swear first thing I'm doing after all this is updating the shared
> code.  Try again please,

It went farther:

processed 163840 of 1064960 possible bytes, 15%
Recording extents for root 5
processed 65536 of 10960896 possible bytes, 0%
Recording extents for root 7
processed 16384 of 16570974208 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 161197
processed 131072 of 108986368 possible bytes, 0%
Recording extents for root 161199
processed 196608 of 49479680 possible bytes, 0%
Recording extents for root 161200
processed 180224 of 254214144 possible bytes, 0%
Recording extents for root 161889
processed 196608 of 49446912 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 162632
processed 114688 of 94633984 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 163302
processed 98304 of 94633984 possible bytes, 0%
Recording extents for root 163303
processed 131072 of 76333056 possible bytes, 0%
Recording extents for root 163316
processed 98304 of 108544000 possible bytes, 0%
Recording extents for root 163920
processed 16384 of 108691456 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 164623
processed 311296 of 63193088 possible bytes, 0%
Recording extents for root 164624
processed 933888 of 108822528 possible bytes, 0%
Recording extents for root 164629
processed 622592 of 108838912 possible bytes, 0%
Recording extents for root 164631
processed 16384 of 49430528 possible bytes, 0%
Recording extents for root 164633
processed 16384 of 75694080 possible bytes, 0%
Recording extents for root 164823
processed 131072 of 63193088 possible bytes, 0%
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes, 100%
doing block accounting
couldn't find a block group at bytenr 20971520 total left 180224
cache 11106814787584 11107888529408 in range no
cache 11108962271232 11110036013056 in range no
cache 11110036013056 11111109754880 in range no
(...)
cache 15929559744512 15930633486336 in range no
cache 15930633486336 15931707228160 in range no
cache 15931707228160 15932780969984 in range no
cache 15932780969984 15933854711808 in range no
cache 15933854711808 15934928453632 in range no
ERROR: update block group failed 20971520 180224 ret -1
FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

doing close???
ERROR: commit_root already set when starting transaction
extent buffer leak: start 15645020913664 len 16384
Init extent tree failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-08  2:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-06-08 20:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-06-08 21:30                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-08 20:57 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Tue, Jun 7, 2022 at 10:12 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Tue, Jun 07, 2022 at 09:31:28PM -0400, Josef Bacik wrote:
> > > doing close???
> > > extent buffer leak: start 11160502779904 len 16384
> > > extent buffer leak: start 15645018308608 len 16384
> > > Init extent tree failed
> >
> > I swear first thing I'm doing after all this is updating the shared
> > code.  Try again please,
>
> It went farther:
>
> processed 163840 of 1064960 possible bytes, 15%
> Recording extents for root 5
> processed 65536 of 10960896 possible bytes, 0%
> Recording extents for root 7
> processed 16384 of 16570974208 possible bytes, 0%
> Recording extents for root 9
> processed 16384 of 16384 possible bytes, 100%
> Recording extents for root 161197
> processed 131072 of 108986368 possible bytes, 0%
> Recording extents for root 161199
> processed 196608 of 49479680 possible bytes, 0%
> Recording extents for root 161200
> processed 180224 of 254214144 possible bytes, 0%
> Recording extents for root 161889
> processed 196608 of 49446912 possible bytes, 0%
> Recording extents for root 162628
> processed 49152 of 49463296 possible bytes, 0%
> Recording extents for root 162632
> processed 114688 of 94633984 possible bytes, 0%
> Recording extents for root 163298
> processed 49152 of 49463296 possible bytes, 0%
> Recording extents for root 163302
> processed 98304 of 94633984 possible bytes, 0%
> Recording extents for root 163303
> processed 131072 of 76333056 possible bytes, 0%
> Recording extents for root 163316
> processed 98304 of 108544000 possible bytes, 0%
> Recording extents for root 163920
> processed 16384 of 108691456 possible bytes, 0%
> Recording extents for root 164620
> processed 49152 of 49463296 possible bytes, 0%
> Recording extents for root 164623
> processed 311296 of 63193088 possible bytes, 0%
> Recording extents for root 164624
> processed 933888 of 108822528 possible bytes, 0%
> Recording extents for root 164629
> processed 622592 of 108838912 possible bytes, 0%
> Recording extents for root 164631
> processed 16384 of 49430528 possible bytes, 0%
> Recording extents for root 164633
> processed 16384 of 75694080 possible bytes, 0%
> Recording extents for root 164823
> processed 131072 of 63193088 possible bytes, 0%
> Recording extents for root 18446744073709551607
> processed 16384 of 16384 possible bytes, 100%
> doing block accounting
> couldn't find a block group at bytenr 20971520 total left 180224
> cache 11106814787584 11107888529408 in range no
> cache 11108962271232 11110036013056 in range no
> cache 11110036013056 11111109754880 in range no
> (...)
> cache 15929559744512 15930633486336 in range no
> cache 15930633486336 15931707228160 in range no
> cache 15931707228160 15932780969984 in range no
> cache 15932780969984 15933854711808 in range no
> cache 15933854711808 15934928453632 in range no
> ERROR: update block group failed 20971520 180224 ret -1
> FIX BLOCK ACCOUNTING FAILED -1

Oops I think this is a system chunk, I've added some code to do the
right thing, can you give this a whirl and see if it fixes it?
Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-08 20:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-06-08 21:30                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-06-08 21:33                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-08 21:30 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 08, 2022 at 04:57:05PM -0400, Josef Bacik wrote:
 
> Oops I think this is a system chunk, I've added some code to do the
> right thing, can you give this a whirl and see if it fixes it?
> Thanks,

Better :)

processed 196608 of 63193088 possible bytes, 0%
searching 18446744073709551607 for bad extents
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 3
processed 180224 of 0 possible bytes, 0%
Recording extents for root 1
processed 999424 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 163840 of 1064960 possible bytes, 15%
Recording extents for root 5
processed 65536 of 10960896 possible bytes, 0%
Recording extents for root 7
processed 16384 of 16570974208 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 161197
processed 131072 of 108986368 possible bytes, 0%
Recording extents for root 161199
processed 196608 of 49479680 possible bytes, 0%
Recording extents for root 161200
processed 180224 of 254214144 possible bytes, 0%
Recording extents for root 161889
processed 196608 of 49446912 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 162632
processed 114688 of 94633984 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 163302
processed 98304 of 94633984 possible bytes, 0%
Recording extents for root 163303
processed 131072 of 76333056 possible bytes, 0%
Recording extents for root 163316
processed 98304 of 108544000 possible bytes, 0%
Recording extents for root 163920
processed 16384 of 108691456 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 164623
processed 311296 of 63193088 possible bytes, 0%
Recording extents for root 164624
processed 933888 of 108822528 possible bytes, 0%
Recording extents for root 164629
processed 622592 of 108838912 possible bytes, 0%
Recording extents for root 164631
processed 16384 of 49430528 possible bytes, 0%
Recording extents for root 164633
processed 16384 of 75694080 possible bytes, 0%
Recording extents for root 164823
processed 131072 of 63193088 possible bytes, 0%
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes, 100%
doing block accounting
doing close???
Init extent tree finished, you can run check now
[Inferior 1 (process 12822) exited normally]


Do I run check --repair now?
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-08 21:30                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-06-08 21:33                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-06-08 21:38                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-08 21:33 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 8, 2022 at 5:30 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Jun 08, 2022 at 04:57:05PM -0400, Josef Bacik wrote:
>
> > Oops I think this is a system chunk, I've added some code to do the
> > right thing, can you give this a whirl and see if it fixes it?
> > Thanks,
>
> Better :)
>
> processed 196608 of 63193088 possible bytes, 0%
> searching 18446744073709551607 for bad extents
> processed 16384 of 16384 possible bytes, 100%
> Recording extents for root 3
> processed 180224 of 0 possible bytes, 0%
> Recording extents for root 1
> processed 999424 of 0 possible bytes, 0%
> doing roots
> Recording extents for root 4
> processed 163840 of 1064960 possible bytes, 15%
> Recording extents for root 5
> processed 65536 of 10960896 possible bytes, 0%
> Recording extents for root 7
> processed 16384 of 16570974208 possible bytes, 0%
> Recording extents for root 9
> processed 16384 of 16384 possible bytes, 100%
> Recording extents for root 161197
> processed 131072 of 108986368 possible bytes, 0%
> Recording extents for root 161199
> processed 196608 of 49479680 possible bytes, 0%
> Recording extents for root 161200
> processed 180224 of 254214144 possible bytes, 0%
> Recording extents for root 161889
> processed 196608 of 49446912 possible bytes, 0%
> Recording extents for root 162628
> processed 49152 of 49463296 possible bytes, 0%
> Recording extents for root 162632
> processed 114688 of 94633984 possible bytes, 0%
> Recording extents for root 163298
> processed 49152 of 49463296 possible bytes, 0%
> Recording extents for root 163302
> processed 98304 of 94633984 possible bytes, 0%
> Recording extents for root 163303
> processed 131072 of 76333056 possible bytes, 0%
> Recording extents for root 163316
> processed 98304 of 108544000 possible bytes, 0%
> Recording extents for root 163920
> processed 16384 of 108691456 possible bytes, 0%
> Recording extents for root 164620
> processed 49152 of 49463296 possible bytes, 0%
> Recording extents for root 164623
> processed 311296 of 63193088 possible bytes, 0%
> Recording extents for root 164624
> processed 933888 of 108822528 possible bytes, 0%
> Recording extents for root 164629
> processed 622592 of 108838912 possible bytes, 0%
> Recording extents for root 164631
> processed 16384 of 49430528 possible bytes, 0%
> Recording extents for root 164633
> processed 16384 of 75694080 possible bytes, 0%
> Recording extents for root 164823
> processed 131072 of 63193088 possible bytes, 0%
> Recording extents for root 18446744073709551607
> processed 16384 of 16384 possible bytes, 100%
> doing block accounting
> doing close???
> Init extent tree finished, you can run check now
> [Inferior 1 (process 12822) exited normally]
>
>
> Do I run check --repair now?

Just check, no --repair.  I want to make sure the only thing that is
missing is the corresponding device extents for the chunks we
recovered.  I'm going to start writing the code to do that now, but if
there's any errors other than missing device extents then we need to
figure out what those problems are and what to do about them.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-08 21:33                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-06-08 21:38                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-06-08 22:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-08 21:38 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 08, 2022 at 05:33:42PM -0400, Josef Bacik wrote:
> 
> Just check, no --repair.  I want to make sure the only thing that is
> missing is the corresponding device extents for the chunks we
> recovered.  I'm going to start writing the code to do that now, but if
> there's any errors other than missing device extents then we need to
> figure out what those problems are and what to do about them.  Thanks,

woah, it ran in less than 1 minute, that's amazing (I remember when it
took days)

Partial output

[1/7] checking root items
checksum verify failed on 15645959372800 wanted 0x847c08bf found 0x17a9e2f1
checksum verify failed on 15645959389184 wanted 0x3cc757a7 found 0x3b4eff03
checksum verify failed on 15645681451008 wanted 0x7516a3d9 found 0x97f7437d
checksum verify failed on 15646003970048 wanted 0xf18cc579 found 0x1bc64584
checksum verify failed on 15645867720704 wanted 0x14cc427a found 0x9f516106
checksum verify failed on 15645529604096 wanted 0xd11e24d5 found 0x8d01bc00
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 15645959356416 wanted 0x2fa8537e found 0x90ac1f4e
(...)
[2/7] checking extents
checksum verify failed on 15645930487808 wanted 0xced0911a found 0x7124e4f9
Chunk[256, 228, 11106814787584] stripe[1, 11121855561728] is not found in dev extent
Chunk[256, 228, 11108962271232] stripe[1, 11124003045376] is not found in dev extent
Chunk[256, 228, 11110036013056] stripe[1, 11125076787200] is not found in dev extent
Chunk[256, 228, 11111109754880] stripe[1, 11126150529024] is not found in dev extent
Chunk[256, 228, 11112183496704] stripe[1, 11127224270848] is not found in dev extent
Chunk[256, 228, 11113257238528] stripe[1, 11128298012672] is not found in dev extent
Chunk[256, 228, 11114330980352] stripe[1, 11129371754496] is not found in dev extent
Chunk[256, 228, 11115404722176] stripe[1, 11130445496320] is not found in dev extent
Chunk[256, 228, 11116478464000] stripe[1, 11131519238144] is not found in dev extent
Chunk[256, 228, 11118625947648] stripe[1, 11133666721792] is not found in dev extent
Chunk[256, 228, 11119699689472] stripe[1, 11134740463616] is not found in dev extent
(...)
Chunk[256, 228, 15931707228160] stripe[1, 14489109921792] is not found in dev extent
Chunk[256, 228, 15932780969984] stripe[1, 14490183663616] is not found in dev extent
Chunk[256, 228, 15933854711808] stripe[1, 14496626114560] is not found in dev extent
Block group[20971520, 8388608] (flags = 34) didn't find the relative chunk.
Device extent[1, 709781094400, 1073741824] didn't find the relative chunk.
Device extent[1, 710854836224, 1073741824] didn't find the relative chunk.
Device extent[1, 711928578048, 1073741824] didn't find the relative chunk.
Device extent[1, 713002319872, 1073741824] didn't find the relative chunk.
(...)
Device extent[1, 14951892647936, 1073741824] didn't find its device.
Device extent[1, 14952966389760, 1073741824] didn't find its device.
Device extent[1, 14954040131584, 1073741824] didn't find its device.
Device extent[1, 14955113873408, 1073741824] didn't find its device.
Device extent[1, 14956187615232, 1073741824] didn't find its device.
Device extent[1, 14957261357056, 1073741824] didn't find its device.
Device extent[1, 14958335098880, 1073741824] didn't find its device.
Device extent[1, 14959408840704, 1073741824] didn't find its device.
Device extent[1, 14960482582528, 1073741824] didn't find its device.
Device extent[1, 14961556324352, 1073741824] didn't find its device.
Device extent[1, 14962630066176, 1073741824] didn't find its device.
Device extent[1, 14963703808000, 1073741824] didn't find its device.
Device extent[1, 14964777549824, 1073741824] didn't find its device.
Device extent[1, 14965851291648, 1073741824] didn't find its device.
Device extent[1, 14966925033472, 1073741824] didn't find its device.
Device extent[1, 14967998775296, 1073741824] didn't find its device.
Device extent[1, 14969072517120, 1073741824] didn't find its device.
Device extent[1, 14970146258944, 1073741824] didn't find its device.
Device extent[1, 14971220000768, 1073741824] didn't find its device.
Device extent[1, 14972293742592, 1073741824] didn't find its device.
Device extent[1, 14973367484416, 1073741824] didn't find its device.
Device extent[1, 14974441226240, 1073741824] didn't find its device.
Device extent[1, 14975514968064, 1073741824] didn't find its device.
Device extent[1, 14976588709888, 1073741824] didn't find its device.
Device extent[1, 14977662451712, 1073741824] didn't find its device.
Device extent[1, 14978736193536, 1073741824] didn't find its device.
Device extent[1, 14979809935360, 1073741824] didn't find its device.
Device extent[1, 14980883677184, 1073741824] didn't find its device.
Device extent[1, 14981957419008, 1073741824] didn't find its device.
Device extent[1, 14983031160832, 1073741824] didn't find its device.
Device extent[1, 14984104902656, 1073741824] didn't find its device.
Device extent[1, 14985178644480, 1073741824] didn't find its device.
Device extent[1, 14986252386304, 1073741824] didn't find its device.
Device extent[1, 14987326128128, 1073741824] didn't find its device.
Device extent[1, 14988399869952, 1073741824] didn't find its device.
Device extent[1, 14989473611776, 1073741824] didn't find its device.
Device extent[1, 14990547353600, 1073741824] didn't find its device.
Device extent[1, 14991621095424, 1073741824] didn't find its device.
Device extent[1, 14992694837248, 1073741824] didn't find its device.
Device extent[1, 14993768579072, 1073741824] didn't find its device.
Device extent[1, 14994842320896, 1073741824] didn't find its device.
Device extent[1, 14995916062720, 1073741824] didn't find its device.
Device extent[1, 14996989804544, 1073741824] didn't find its device.
Device extent[1, 14998063546368, 1073741824] didn't find its device.
Device extent[1, 14999137288192, 1073741824] didn't find its device.
Device extent[1, 15000211030016, 1073741824] didn't find its device.
Device extent[1, 15001284771840, 1073741824] didn't find its device.
Device extent[1, 15002358513664, 1073741824] didn't find its device.
Device extent[1, 15003432255488, 1073741824] didn't find its device.
Device extent[1, 15004505997312, 1073741824] didn't find its device.
Device extent[1, 15005579739136, 1073741824] didn't find its device.
Device extent[1, 15006653480960, 1073741824] didn't find its device.
Device extent[1, 15007727222784, 1073741824] didn't find its device.
Device extent[1, 15008800964608, 1073741824] didn't find its device.
Device extent[1, 15009874706432, 1073741824] didn't find its device.
Device extent[1, 15010948448256, 1073741824] didn't find its device.
Device extent[1, 15012022190080, 1073741824] didn't find its device.
Device extent[1, 15013095931904, 1073741824] didn't find its device.
Device extent[1, 15014169673728, 1073741824] didn't find its device.
        unresolved ref dir 55138 index 46 namelen 26 name foo filetype 1 errors 5, no dir item, no inode ref 
root 161199 inode 55184 errors 1, no inode item
        unresolved ref dir 55138 index 47 namelen 47 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55185 errors 1, no inode item 
        unresolved ref dir 55138 index 48 namelen 45 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55186 errors 1, no inode item 
        unresolved ref dir 55138 index 49 namelen 42 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55187 errors 1, no inode item 
        unresolved ref dir 55138 index 50 namelen 51 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55188 errors 1, no inode item
        unresolved ref dir 55138 index 51 namelen 47 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55403 errors 2500, file extent discount, nbytes wrong, link count wrong
root 161889 inode 70800 errors 2001, no inode item, link count wrong
        unresolved ref dir 3327 index 33 namelen 15 name foo filetype 2 errors 4, no inode ref
root 161889 inode 71665 errors 2001, no inode item, link count wrong 
        unresolved ref dir 3327 index 35 namelen 13 name foo filetype 2 errors 4, no inode ref
root 161889 inode 71860 errors 2001, no inode item, link count wrong
        unresolved ref dir 3327 index 36 namelen 41 name foo filetype 2 errors 4, no inode ref
root 161889 inode 71905 errors 2001, no inode item, link count wrong
        unresolved ref dir 3327 index 37 namelen 69 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72375 errors 2001, no inode item, link count wrong
        unresolved ref dir 3327 index 38 namelen 14 name foo filetype 2 errors 4, no inode ref 
root 161889 inode 72543 errors 2001, no inode item, link count wrong
        unresolved ref dir 3327 index 39 namelen 12 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72593 errors 2001, no inode item, link count wrong
        unresolved ref dir 3327 index 40 namelen 4 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72723 errors 2001, no inode item, link count wrong 
        unresolved ref dir 3327 index 45 namelen 15 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72724 errors 2001, no inode item, link count wrong
        unresolved ref dir 3327 index 46 namelen 29 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72738 errors 2001, no inode item, link count wrong
        unresolved ref dir 3327 index 47 namelen 25 name foo filetype 2 errors 4, no inode ref 
root 161889 inode 72779 errors 2001, no inode item, link count wrong
        unresolved ref dir 3327 index 48 namelen 33 name foo filetype 1 errors 4, no inode ref
root 161889 inode 72780 errors 2001, no inode item, link count wrong
        unresolved ref dir 3327 index 49 namelen 33 name foo filetype 1 errors 4, no inode ref
root 161889 inode 72781 errors 2001, no inode item, link count wrong 
        unresolved ref dir 3327 index 50 namelen 33 name foo filetype 1 errors 4, no inode ref
root 161889 inode 72782 errors 2001, no inode item, link count wrong
        unresolved ref dir 3327 index 51 namelen 21 name foo filetype 1 errors 4, no inode ref
root 161889 inode 72783 errors 2001, no inode item, link count wrong
        unresolved ref dir 3327 index 52 namelen 40 name foo filetype 1 errors 4, no inode ref 
root 161889 inode 72784 errors 2001, no inode item, link count wrong
        unresolved ref dir 3327 index 53 namelen 40 name foo filetype 1 errors 4, no inode ref
root 162628 root dir 256 not found
root 162628 inode 49389 errors 2500, file extent discount, nbytes wrong, link count wrong
index, no inode ref
root 163302 inode 11222 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 43 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11232 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 75 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11233 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 41 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11239 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 82 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11240 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 51 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11246 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 56 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11249 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 51 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11266 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 53 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11271 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 55 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11275 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 42 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11276 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11278 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 48 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11285 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 58 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11293 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 51 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11296 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 41 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11301 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 55 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11306 errors 2001, no inode item, link count wrong
        unresolved ref dir 11179 index 0 namelen 49 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11307 errors 2001, no inode item, link count wrong 
        unresolved ref dir 11179 index 0 namelen 60 name foo filetype 1 errorFound file extent holes:
        start: 0, len: 260874240
root 164623 inode 3682 errors 500, file extent discount, nbytes wrong
Found file extent holes:
        start: 0, len: 103227392
root 164623 inode 3683 errors 500, file extent discount, nbytes wrong
Found file extent holes:
        start: 0, len: 58925056
root 164623 inode 3684 errors 500, file extent discount, nbytes wrong
Found file extent holes:
        start: 0, len: 22671360
root 164623 inode 3685 errors 500, file extent discount, nbytes wrong
Found file extent holes:
        start: 0, len: 55967744
root 164623 inode 3686 errors 500, file extent discount, nbytes wrong
Found file extent holes:
        start: 0, len: 164442112
        unresolved ref dir 3676 index 11 namelen 62 name foo filetype 1 errroot 164629 inode 859 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 65 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 860 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 63 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 861 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 66 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 862 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 59 name foo filetype 1 errors 6, no dir index, no inode ref 
root 164629 inode 864 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 72 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 865 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 58 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 866 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 88 name foo filetype 7 errors 6, no dir index, no inode ref
root 164629 inode 867 errors 2001, no inode item, link count wrong 
        unresolved ref dir 842 index 0 namelen 57 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 868 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 70 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 872 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 101 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 873 errors 2001, no inode item, link count wrong 
        unresolved ref dir 842 index 0 namelen 80 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 874 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 69 name foo filetype 1 errors 6, no dir index, no inode ref 
root 164629 inode 875 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 79 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 876 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 51 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 877 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 95 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 878 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 879 errors 2001, no inode item, link count wrong 
        unresolved ref dir 842 index 0 namelen 75 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 880 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 44 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 881 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 67 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 884 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 70 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 885 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 84 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 887 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 888 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 74 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 890 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 80 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 891 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 892 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 91 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 904 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 79 name foo filetype 1 errors 6, no dir index, no inode ref 
root 164629 inode 905 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 101 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 907 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 62 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 909 errors 2001, no inode item, link count wrong 
        unresolved ref dir 842 index 0 namelen 70 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 910 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 74 name foo filetype 1 errors 6, no dir index, no inode ref 
root 164629 inode 911 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 63 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 912 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 84 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 913 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 93 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 914 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 68 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 917 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 79 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 918 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 89 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 921 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 922 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 72 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 923 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 925 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 80 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 926 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 70 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 927 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 70 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 928 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 72 name foo filetype 7 errors 6, no dir index, no inode ref
root 164629 inode 1047 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 86 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1050 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 88 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1051 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 81 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1052 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 91 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1053 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 57 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1054 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 67 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1055 errors 2001, no inode item, link count wrong
        unresolved ref dir 842 index 0 namelen 78 name foo froot 164629 inode 17822 errors 2001, no inode item, link count wrong
        unresolved ref dir 791 index 144 namelen 58 name foo filetype 1 errors 4, no inode ref
root 164629 inode 17824 errors 1, no inode item
        unresolved ref dir 791 index 145 namelen 47 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17893 errors 1, no inode item
root 164629 inode 17894 errors 2500, file extent discount, nbytes wrong, link count wrong 
Found file extent holes:
        start: 0, len: 9650176
        unresolved ref dir 17893 index 2 namelen 50 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 17895 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes: 
        start: 0, len: 64696320
root 164629 inode 17896 errors 1, no inode item
        unresolved ref dir 17893 index 4 namelen 57 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17897 errors 1, no inode item
        unresolved ref dir 17893 index 5 namelen 26 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17898 errors 1, no inode item
        unresolved ref dir 17893 index 6 namelen 31 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17899 errors 1, no inode item
        unresolved ref dir 17893 index 7 namelen 46 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17900 errors 1, no inode item
        unresolved ref dir 17893 index 8 namelen 119 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17901 errors 1, no inode item
        unresolved ref dir 17893 index 9 namelen 66 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17902 errors 1, no inode item
        unresolved ref dir 17893 index 10 namelen 44 name foo filetype 1 errors 5, no dir item, no inode ref 
root 164629 inode 17903 errors 1, no inode item
        unresolved ref dir 17893 index 11 namelen 111 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17904 errors 1, no inode item
        unresolved ref dir 17893 index 12 namelen 34 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17905 errors 1, no inode item
        unresolved ref dir 17893 index 13 namelen 34 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17906 errors 1, no inode item
        unresolved ref dir 17893 index 14 namelen 41 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17907 errors 1, no inode item
        unresolved ref dir 17893 index 15 namelen 59 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17908 errors 1, no inode item
        unresolved ref dir 17893 index 16 namelen 47 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17909 errors 1, no inode item 
        unresolved ref dir 17893 index 17 namelen 25 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17910 errors 1, no inode item
root 164823 inode 25244 errors 2001, no inode item, link count wrong
        unresolved ref dir 3654 index 34 namelen 9 name foo filetype 2 errors 4, no inode ref
root 164823 inode 25246 errors 2001, no inode item, link count wrong 
        unresolved ref dir 842 index 296 namelen 70 name foo filetype 1 errors 4, no inode ref
root 164823 inode 27724 errors 2001, no inode item, link count wrong 
        unresolved ref dir 3654 index 35 namelen 5 name foo filetype 2 errors 4, no inode ref
root 164823 inode 34022 errors 2001, no inode item, link count wrong
        unresolved ref dir 3654 index 39 namelen 3 name foo filetype 2 errors 4, no inode ref 
root 164823 inode 34023 errors 2001, no inode item, link count wrong
        unresolved ref dir 3654 index 40 namelen 3 name foo filetype 2 errors 4, no inode ref
root 164823 inode 34454 errors 2001, no inode item, link count wrong
        unresolved ref dir 3654 index 60 namelen 11 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34455 errors 2001, no inode item, link count wrong 
        unresolved ref dir 3654 index 62 namelen 6 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34471 errors 2001, no inode item, link count wrong
        unresolved ref dir 3654 index 76 namelen 11 name foo filetype 1 errors 4, no inode ref 
root 164823 inode 34480 errors 2001, no inode item, link count wrong
        unresolved ref dir 3654 index 86 namelen 6 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34482 errors 2001, no inode item, link count wrong 
        unresolved ref dir 3654 index 87 namelen 6 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34508 errors 2001, no inode item, link count wrong
        unresolved ref dir 3654 index 98 namelen 11 name foo filetype 1 errors 4, no inode ref 
root 164823 inode 34511 errors 2001, no inode item, link count wrong
        unresolved ref dir 3654 index 102 namelen 6 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34512 errors 2001, no inode item, link count wrong 
        unresolved ref dir 3654 index 103 namelen 11 name foo filetype 1 errors 4, no inode ref
root 164823 inode 35040 errors 2001, no inode item, link count wrong
        unresolved ref dir 3654 index 159 namelen 11 name foo filetype 1 errors 4, no inode ref 
root 164823 inode 35142 errors 2001, no inode item, link count wrong
        unresolved ref dir 3654 index 209 namelen 6 name foo filetype 1 errors 4, no inode ref
root 164823 inode 35200 errors 1, no inode item 
        unresolved ref dir 791 index 156 namelen 41 name foo filetype 1 errors 5, no dir item, no inode ref
root 164823 inode 35232 errors 2001, no inode item, link count wrong
        unresolved ref dir 3676 index 24 namelen 45 name foo filetype 1 errors 4, no inode ref
root 164823 inode 35241 errors 2001, no inode item, link count wrong
        unresolved ref dir 3676 index 25 namelen 62 name foo filetype 1 errors 4, no inode ref
root 164823 inode 36759 errors 1, no inode item
        unresolved ref dir 791 index 157 namelen 44 name foo filetype 1 errors 5, no dir item, no inode ref
root 164823 inode 37194 errors 2001, no inode item, link count wrong
        unresolved ref dir 3654 index 213 namelen 3 name foo filetype 2 errors 4, no inode ref
root 164823 inode 38289 errors 2001, no inode item, link count wrong
        unresolved ref dir 3654 index 212 namelen 1 name foo filetype 7 errors 4, no inode ref
root 164823 inode 39940 errors 2001, no inode item, link count wrong
        unresolved ref dir 3676 index 27 namelen 103 name foo filetype 1 errors 4, no inode ref
root 164823 inode 39950 errors 2001, no inode item, link count wrong
        unresolved ref dir 3676 index 26 namelen 117 name foo filetype 1 errors 4, no inode ref
root 164823 inode 39951 errors 2001, no inode item, link count wrong
        unresolved ref dir 3676 index 28 namelen 66 name foo filetype 1 errors 4, no inode ref
root 164823 inode 69105 errors 1, no inode item
        unresolved ref dir 68739 index 151 namelen 66 name foo filetype 0 errors 3, no dir item, no dir index
root 164823 inode 69135 errors 2000, link count wrong
        unresolved ref dir 256 index 68892 namelen 4 name foo filetype 0 errors 3, no dir item, no dir index
root 164823 inode 69136 errors 2200, dir isize wrong, link count wrong
        unresolved ref dir 1938 index 4 namelen 40 name foo filetype 0 errors 3, no dir item, no dir index
root 164823 inode 69252 errors 2001, no inode item, link count wrong
        unresolved ref dir 69136 index 0 namelen 31 name foo filetype 2 errors 6, no dir index, no inode ref
root 164823 inode 74108 errors 2001, no inode item, link count wrong
        unresolved ref dir 3280 index 12 namelen 44 name foo filetype 1 errors 4, no inode ref
root 164823 inode 74132 errors 2001, no inode item, link count wrong
        unresolved ref dir 256 index 19071 namelen 4 name foo filetype 2 errors 4, no inode ref
root 164823 inode 74193 errors 2001, no inode item, link count wrong
        unresolved ref dir 3654 index 214 namelen 4 name foo filetype 2 errors 4, no inode ref
root 164823 inode 74838 errors 2001, no inode item, link count wrong
        unresolved ref dir 3280 index 13 namelen 21 name foo filetype 2 errors 4, no inode ref
root 164823 inode 76221 errors 2001, no inode item, link count wrong
        unresolved ref dir 791 index 161 namelen 41 name foo filetype 1 errors 4, no inode ref
root 164823 inode 76328 errors 2001, no inode item, link count wrong
        unresolved ref dir 3280 index 16 namelen 88 name foo filetype 1 errors 4, no inode ref
root 164823 inode 76329 errors 2001, no inode item, link count wrong
        unresolved ref dir 3280 index 17 namelen 88 name foo filetype 1 errors 4, no inode ref
ERROR: errors found in fs roots
found 21912809472 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 5373952
total fs tree bytes: 3424256
total extent tree bytes: 573440
btree space waste bytes: 1834949
file data blocks allocated: 33614336000
 referenced 33612742656

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-08 21:38                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-06-08 22:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-09  3:01                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-08 22:46 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 8, 2022 at 5:38 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Jun 08, 2022 at 05:33:42PM -0400, Josef Bacik wrote:
> >
> > Just check, no --repair.  I want to make sure the only thing that is
> > missing is the corresponding device extents for the chunks we
> > recovered.  I'm going to start writing the code to do that now, but if
> > there's any errors other than missing device extents then we need to
> > figure out what those problems are and what to do about them.  Thanks,
>
> woah, it ran in less than 1 minute, that's amazing (I remember when it
> took days)
>

Ok I've added some stuff to fix the device extents.  Go ahead and run
with --repair and lets see how that goes.  After that finishes run
again without --repair so we can see what's still broken, I imagine
I'll have to clean some other stuff up.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-08 22:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-06-09  3:01                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-09 20:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-09  3:01 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 08, 2022 at 06:46:53PM -0400, Josef Bacik wrote:
> Ok I've added some stuff to fix the device extents.  Go ahead and run
> with --repair and lets see how that goes.  After that finishes run
> again without --repair so we can see what's still broken, I imagine
> I'll have to clean some other stuff up.  Thanks,

Can't determine the filetype for inode 69105, assume it is a normal file
Can't get file type for inode 69105, using FILE as fallback
Moving file f to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 69105
Moving file f to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 69135
reset isize for dir 69136 root 164823
Moving file f to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 69136
Trying to rebuild inode:69252
Moving file f to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 69252
Trying to rebuild inode:74108
Moving file f to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 74108
Trying to rebuild inode:74132
Moving file f to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 74132
Trying to rebuild inode:74193
Moving file f to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 74193
Trying to rebuild inode:74838
Moving file f to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 74838
Trying to rebuild inode:76221
Moving file f to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 76221
Trying to rebuild inode:76328
Moving file f to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 76328
Trying to rebuild inode:76329
Moving file f to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 76329
found 43825717248 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 10846208
total fs tree bytes: 6848512
total extent tree bytes: 1146880
btree space waste bytes: 3705949
file data blocks allocated: 67228672000
 referenced 67225485312
gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro,recovery /dev/mapper/dshelf1 /mnt/mnt
mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error.
gargamel:/var/local/src/btrfs-progs-josefbacik# dmtail
[3890613.672704] BTRFS info (device dm-1): flagging fs with big metadata feature
[3890613.694891] BTRFS warning (device dm-1): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
[3890613.759915] BTRFS info (device dm-1): trying to use backup root at mount time
[3890613.782884] BTRFS info (device dm-1): disk space caching is enabled
[3890613.802960] BTRFS info (device dm-1): has skinny extents
[3890613.826455] BTRFS error (device dm-1): super_num_devices 1 mismatch with num_devices 0 found here
[3890613.855092] BTRFS error (device dm-1): failed to read chunk tree: -22
[3890613.876716] BTRFS error (device dm-1): open_ctree failed

btrfs check ends with:
root 164624 inode 73099 errors 1400, nbytes wrong, some csum missing
root 164624 inode 73100 errors 1400, nbytes wrong, some csum missing
        unresolved ref dir 791 index 0 namelen 25 name f filetype 1 errors 6, no dir index, no inode ref
        unresolved ref dir 3676 index 0 namelen 62 name f filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 3752 errors 200, dir isize wrong
        unresolved ref dir 73103 index 540 namelen 4 name f filetype mismatch
root 164629 inode 3965 errors 200, dir isize wrong
        unresolved ref dir 4179 index 0 namelen 10 name f filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 4549 errors 200, dir isize wrong
        unresolved ref dir 4698 index 0 namelen 69 name f filetype 1 errors 6, no dir index, no inode ref
        unresolved ref dir 5506 index 0 namelen 53 name f filetype 1 errors 6, no dir index, no inode ref
        unresolved ref dir 5546 index 0 namelen 57 name f filetype 1 errors 6, no dir index, no inode ref 
root 164629 inode 39921 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong
Found file extent holes:
        start: 759824384, len: 2143326208
        unresolved ref dir 10205 index 356 namelen 34 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 40537 errors 1400, nbytes wrong, some csum missing
root 164629 inode 72418 errors 1400, nbytes wrong, some csum missing
root 164629 inode 72429 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong
Found file extent holes:
        start: 1333788672, len: 3217154048
        unresolved ref dir 72438 index 4 namelen 49 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72433 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong
Found file extent holes:
        start: 892338176, len: 14957748224
        unresolved ref dir 34951 index 13 namelen 46 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72587 errors 200, dir isize wrong
root 164629 inode 72588 errors 1000, some csum missing
root 164629 inode 72592 errors 200, dir isize wrong
root 164629 inode 72593 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong
Found file extent holes:
        start: 42074112, len: 413642752
        unresolved ref dir 75036 index 19 namelen 45 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72639 errors 200, dir isize wrong
root 164629 inode 72640 errors 200, dir isize wrong
root 164629 inode 72672 errors 200, dir isize wrong
        unresolved ref dir 73103 index 1386 namelen 5 name f filetype mismatch
root 164629 inode 73001 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73006 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73009 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong
Found file extent holes:
        start: 13869056, len: 53100544
        unresolved ref dir 72672 index 134 namelen 26 name f filetype 1 errors 1, no dir item
root 164629 inode 73045 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73066 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73067 errors 1000, some csum missing
root 164629 inode 73082 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73083 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong
Found file extent holes:
        start: 197263360, len: 399777792 
        unresolved ref dir 3747 index 30 namelen 70 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 73086 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73094 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong
Found file extent holes:
        start: 0, len: 524288 
        unresolved ref dir 74963 index 32 namelen 56 name f filetype 0 errors 3, no dir item, no dir index 
root 164629 inode 73097 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong
Found file extent holes: 
        start: 0, len: 524288
        unresolved ref dir 74963 index 36 namelen 56 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 73099 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73100 errors 1400, nbytes wrong, some csum missing
        unresolved ref dir 791 index 0 namelen 25 name f filetype 1 errors 6, no dir index, no inode ref
        unresolved ref dir 3676 index 0 namelen 62 name f filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs roots
Opening filesystem to check...
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
cache and super generation don't match, space cache will be invalidated
found 21916315648 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8880128
total fs tree bytes: 6799360
total extent tree bytes: 606208
btree space waste bytes: 2518033
file data blocks allocated: 36729012224
 referenced 36727418880

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-09  3:01                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-09 20:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-06-09 21:15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-09 20:46 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 8, 2022 at 11:01 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Jun 08, 2022 at 06:46:53PM -0400, Josef Bacik wrote:
> > Ok I've added some stuff to fix the device extents.  Go ahead and run
> > with --repair and lets see how that goes.  After that finishes run
> > again without --repair so we can see what's still broken, I imagine
> > I'll have to clean some other stuff up.  Thanks,
>
> Can't determine the filetype for inode 69105, assume it is a normal file
> Can't get file type for inode 69105, using FILE as fallback
> Moving file f to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 69105
> Moving file f to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 69135
> reset isize for dir 69136 root 164823
> Moving file f to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 69136
> Trying to rebuild inode:69252
> Moving file f to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 69252
> Trying to rebuild inode:74108
> Moving file f to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 74108
> Trying to rebuild inode:74132
> Moving file f to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 74132
> Trying to rebuild inode:74193
> Moving file f to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 74193
> Trying to rebuild inode:74838
> Moving file f to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 74838
> Trying to rebuild inode:76221
> Moving file f to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 76221
> Trying to rebuild inode:76328
> Moving file f to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 76328
> Trying to rebuild inode:76329
> Moving file f to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 76329
> found 43825717248 bytes used, error(s) found
> total csum bytes: 0
> total tree bytes: 10846208
> total fs tree bytes: 6848512
> total extent tree bytes: 1146880
> btree space waste bytes: 3705949
> file data blocks allocated: 67228672000
>  referenced 67225485312
> gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro,recovery /dev/mapper/dshelf1 /mnt/mnt
> mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error.
> gargamel:/var/local/src/btrfs-progs-josefbacik# dmtail
> [3890613.672704] BTRFS info (device dm-1): flagging fs with big metadata feature
> [3890613.694891] BTRFS warning (device dm-1): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
> [3890613.759915] BTRFS info (device dm-1): trying to use backup root at mount time
> [3890613.782884] BTRFS info (device dm-1): disk space caching is enabled
> [3890613.802960] BTRFS info (device dm-1): has skinny extents
> [3890613.826455] BTRFS error (device dm-1): super_num_devices 1 mismatch with num_devices 0 found here
> [3890613.855092] BTRFS error (device dm-1): failed to read chunk tree: -22
> [3890613.876716] BTRFS error (device dm-1): open_ctree failed

Sorry this took me longer than normal to work out what happened and
how to fix it properly.

You can pull and re-run btrfs check --repair <device> and it should
fix the device thing that's missing.  We can tackle the other fs
errors next, but I want to see if we can at least get you mounting the
fs.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-09 20:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-06-09 21:15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-06-10 18:47                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-09 21:15 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Thu, Jun 09, 2022 at 04:46:28PM -0400, Josef Bacik wrote:
> Sorry this took me longer than normal to work out what happened and
> how to fix it properly.
 
It takes the time it takes, I'm sure nothing there is obvious

> You can pull and re-run btrfs check --repair <device> and it should
> fix the device thing that's missing.  We can tackle the other fs
> errors next, but I want to see if we can at least get you mounting the
> fs.  Thanks,

Done:
WARNING: reserved space leaked, transid=2606858 flag=0x2 bytes_reserved=49152
root 164629 inode 73099 errors 1000, some csum missing
WARNING: reserved space leaked, transid=2606859 flag=0x2 bytes_reserved=49152
root 164629 inode 73100 errors 1000, some csum missing
        unresolved ref dir 791 index 0 namelen 25 name Banlieue 13 Ultimatum.avi filetype 1 errors 6, no dir index, no inode ref
        unresolved ref dir 3676 index 0 namelen 62 name foo filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs roots
WARNING: reserved space leaked, flag=0x2 bytes_reserved=49152
extent buffer leak: start 15645018226688 len 16384

Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
cache and super generation don't match, space cache will be invalidated
Fixed the nlink of inode 5644
reset nbytes for ino 54987 root 161199
reset nbytes for ino 54997 root 161199
reset nbytes for ino 55002 root 161199
reset isize for dir 55138 root 161199
reset nbytes for ino 55458 root 161199
reset nbytes for ino 55467 root 161199
reset nbytes for ino 55474 root 161199
reset nbytes for ino 55525 root 161199
reset nbytes for ino 54987 root 161889
reset nbytes for ino 54997 root 161889
reset nbytes for ino 55002 root 161889
reset isize for dir 55138 root 161889
reset nbytes for ino 55458 root 161889
reset nbytes for ino 55467 root 161889
reset nbytes for ino 55474 root 161889
reset nbytes for ino 55525 root 161889
Fixed the nlink of inode 95666
Fixed the nlink of inode 95666
Fixed the nlink of inode 95666
reset isize for dir 4549 root 164624
reset isize for dir 25810 root 164624
reset isize for dir 25812 root 164624
reset nbytes for ino 26004 root 164624
reset isize for dir 31346 root 164624
reset nbytes for ino 72418 root 164624
reset isize for dir 72587 root 164624
reset isize for dir 72592 root 164624
reset isize for dir 72639 root 164624
reset isize for dir 72640 root 164624
reset isize for dir 72672 root 164624
reset nbytes for ino 73001 root 164624
reset nbytes for ino 73006 root 164624
reset nbytes for ino 73045 root 164624
reset nbytes for ino 73066 root 164624
reset nbytes for ino 73082 root 164624
reset nbytes for ino 73086 root 164624
reset nbytes for ino 73099 root 164624
reset nbytes for ino 73100 root 164624
reset isize for dir 3747 root 164629
reset isize for dir 3752 root 164629
reset isize for dir 3965 root 164629
reset isize for dir 4549 root 164629
reset nbytes for ino 40537 root 164629
reset nbytes for ino 72418 root 164629
reset isize for dir 72587 root 164629
reset isize for dir 72592 root 164629
reset isize for dir 72639 root 164629
reset isize for dir 72640 root 164629
reset isize for dir 72672 root 164629
reset nbytes for ino 73001 root 164629
reset nbytes for ino 73006 root 164629
reset nbytes for ino 73045 root 164629
reset nbytes for ino 73066 root 164629
reset nbytes for ino 73082 root 164629
reset nbytes for ino 73086 root 164629
reset nbytes for ino 73099 root 164629
reset nbytes for ino 73100 root 164629
found 21916315648 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8880128
total fs tree bytes: 6799360
total extent tree bytes: 606208
btree space waste bytes: 2518033
file data blocks allocated: 36729012224
 referenced 36727418880
gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro,recovery /dev/mapper/dshelf1 /mnt/mnt
mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error

[3956372.768821] BTRFS info (device dm-1): disk space caching is enabled
[3956372.789085] BTRFS info (device dm-1): has skinny extents
[3956372.863763] BTRFS error (device dm-1): dev extent physical offset 709781094400 on devid 1 doesn't have corresponding chunk
[3956372.899452] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117
[3956372.926355] BTRFS error (device dm-1): open_ctree failed
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-09 21:15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-06-10 18:47                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  2022-06-10 19:11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-10 18:47 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Thu, Jun 9, 2022 at 5:15 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Jun 09, 2022 at 04:46:28PM -0400, Josef Bacik wrote:
> > Sorry this took me longer than normal to work out what happened and
> > how to fix it properly.
>
> It takes the time it takes, I'm sure nothing there is obvious
>
> > You can pull and re-run btrfs check --repair <device> and it should
> > fix the device thing that's missing.  We can tackle the other fs
> > errors next, but I want to see if we can at least get you mounting the
> > fs.  Thanks,
>
> Done:
> WARNING: reserved space leaked, transid=2606858 flag=0x2 bytes_reserved=49152
> root 164629 inode 73099 errors 1000, some csum missing
> WARNING: reserved space leaked, transid=2606859 flag=0x2 bytes_reserved=49152
> root 164629 inode 73100 errors 1000, some csum missing
>         unresolved ref dir 791 index 0 namelen 25 name Banlieue 13 Ultimatum.avi filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 3676 index 0 namelen 62 name foo filetype 1 errors 6, no dir index, no inode ref
> ERROR: errors found in fs roots
> WARNING: reserved space leaked, flag=0x2 bytes_reserved=49152
> extent buffer leak: start 15645018226688 len 16384
>
> Starting repair.
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/dshelf1
> UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
> cache and super generation don't match, space cache will be invalidated
> Fixed the nlink of inode 5644
> reset nbytes for ino 54987 root 161199
> reset nbytes for ino 54997 root 161199
> reset nbytes for ino 55002 root 161199
> reset isize for dir 55138 root 161199
> reset nbytes for ino 55458 root 161199
> reset nbytes for ino 55467 root 161199
> reset nbytes for ino 55474 root 161199
> reset nbytes for ino 55525 root 161199
> reset nbytes for ino 54987 root 161889
> reset nbytes for ino 54997 root 161889
> reset nbytes for ino 55002 root 161889
> reset isize for dir 55138 root 161889
> reset nbytes for ino 55458 root 161889
> reset nbytes for ino 55467 root 161889
> reset nbytes for ino 55474 root 161889
> reset nbytes for ino 55525 root 161889
> Fixed the nlink of inode 95666
> Fixed the nlink of inode 95666
> Fixed the nlink of inode 95666
> reset isize for dir 4549 root 164624
> reset isize for dir 25810 root 164624
> reset isize for dir 25812 root 164624
> reset nbytes for ino 26004 root 164624
> reset isize for dir 31346 root 164624
> reset nbytes for ino 72418 root 164624
> reset isize for dir 72587 root 164624
> reset isize for dir 72592 root 164624
> reset isize for dir 72639 root 164624
> reset isize for dir 72640 root 164624
> reset isize for dir 72672 root 164624
> reset nbytes for ino 73001 root 164624
> reset nbytes for ino 73006 root 164624
> reset nbytes for ino 73045 root 164624
> reset nbytes for ino 73066 root 164624
> reset nbytes for ino 73082 root 164624
> reset nbytes for ino 73086 root 164624
> reset nbytes for ino 73099 root 164624
> reset nbytes for ino 73100 root 164624
> reset isize for dir 3747 root 164629
> reset isize for dir 3752 root 164629
> reset isize for dir 3965 root 164629
> reset isize for dir 4549 root 164629
> reset nbytes for ino 40537 root 164629
> reset nbytes for ino 72418 root 164629
> reset isize for dir 72587 root 164629
> reset isize for dir 72592 root 164629
> reset isize for dir 72639 root 164629
> reset isize for dir 72640 root 164629
> reset isize for dir 72672 root 164629
> reset nbytes for ino 73001 root 164629
> reset nbytes for ino 73006 root 164629
> reset nbytes for ino 73045 root 164629
> reset nbytes for ino 73066 root 164629
> reset nbytes for ino 73082 root 164629
> reset nbytes for ino 73086 root 164629
> reset nbytes for ino 73099 root 164629
> reset nbytes for ino 73100 root 164629
> found 21916315648 bytes used, error(s) found
> total csum bytes: 0
> total tree bytes: 8880128
> total fs tree bytes: 6799360
> total extent tree bytes: 606208
> btree space waste bytes: 2518033
> file data blocks allocated: 36729012224
>  referenced 36727418880
> gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro,recovery /dev/mapper/dshelf1 /mnt/mnt
> mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error
>
> [3956372.768821] BTRFS info (device dm-1): disk space caching is enabled
> [3956372.789085] BTRFS info (device dm-1): has skinny extents
> [3956372.863763] BTRFS error (device dm-1): dev extent physical offset 709781094400 on devid 1 doesn't have corresponding chunk
> [3956372.899452] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117
> [3956372.926355] BTRFS error (device dm-1): open_ctree failed

Sorry I keep going back and forth on how to deal with this.  I've
pushed some code, you can run btrfs check --repair again and then try
again.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-10 18:47                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
@ 2022-06-10 19:11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
  2022-06-10 19:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-10 19:11 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Fri, Jun 10, 2022 at 02:47:54PM -0400, Josef Bacik wrote:
> Sorry I keep going back and forth on how to deal with this.  I've
> pushed some code, you can run btrfs check --repair again and then try
> again.  Thanks,

Thanks, lots of output, not sure how much you want.

For one:
Device extent[1, 14803716276224, 1073741824] didn't find the relative chunk.
Device extent[1, 14804790018048, 1073741824] didn't find the relative chunk.
Device extent[1, 14805863759872, 1073741824] didn't find the relative chunk.
Device extent[1, 14806937501696, 1073741824] didn't find the relative chunk.
Device extent[1, 14808011243520, 1073741824] didn't find the relative chunk.
super bytes used 21916233728 mismatches actual used 21916332032
Block group[20971520, 8388608] (flags = 34) didn't find the relative chunk.
Dev extent's total-byte(1396938113024) is not equal to byte-used(14599692746752) in dev[1, 216, 1]
kernel-shared/extent-tree.c:2479: alloc_reserved_tree_block: Warning: assertion `1` failed, value 1
./btrfs(btrfs_run_delayed_refs+0x707)[0x5577947c927b]
./btrfs(btrfs_commit_transaction+0x3b)[0x5577947d7fb5]
./btrfs(repair_dev_item_bytes_used+0x6f)[0x5577948340d7]
./btrfs(+0x7d90b)[0x55779481690b]
./btrfs(+0x80193)[0x557794819193]
./btrfs(main+0x94)[0x5577947b1275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f51c74847fd]
./btrfs(_start+0x2a)[0x5577947b0e1a]
kernel-shared/extent-tree.c:2479: alloc_reserved_tree_block: Warning: assertion `1` failed, value 1
./btrfs(btrfs_run_delayed_refs+0x707)[0x5577947c927b]
./btrfs(btrfs_commit_transaction+0x3b)[0x5577947d7fb5]
./btrfs(repair_dev_item_bytes_used+0x6f)[0x5577948340d7]
./btrfs(+0x7d90b)[0x55779481690b]
./btrfs(+0x80193)[0x557794819193]
./btrfs(main+0x94)[0x5577947b1275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f51c74847fd]
./btrfs(_start+0x2a)[0x5577947b0e1a]
WARNING: reserved space leaked, transid=2608372 flag=0x2 bytes_reserved=32768
[3/7] checking free space cache
[4/7] checking fs roots
root 161199 inode 54987 errors 1000, some csum missing
WARNING: reserved space leaked, transid=2608373 flag=0x2 bytes_reserved=32768
root 161199 inode 54988 errors 1100, file extent discount, some csum missing
Found file extent holes:
        start: 0, len: 1572864
WARNING: reserved space leaked, transid=2608374 flag=0x2 bytes_reserved=32768
root 161199 inode 54989 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong
Found file extent holes:
        start: 16252928, len: 91824128
        unresolved ref dir 54974 index 16 namelen 18 name foo

(...)

Found file extent holes:
        start: 197263360, len: 399777792
root 164629 inode 73086 errors 1000, some csum missing
WARNING: reserved space leaked, transid=2608425 flag=0x2 bytes_reserved=32768
root 164629 inode 73094 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong
Found file extent holes:
        start: 0, len: 524288
        unresolved ref dir 74963 index 32 namelen 56 name file filetype 0 errors 3
, no dir item, no dir index
WARNING: reserved space leaked, transid=2608426 flag=0x2 bytes_reserved=32768
root 164629 inode 73097 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong
Found file extent holes:
        start: 0, len: 524288
        unresolved ref dir 74963 index 36 namelen 56 name file filetype 0 errors 3
, no dir item, no dir index
root 164629 inode 73099 errors 1000, some csum missing
root 164629 inode 73100 errors 1000, some csum missing
        unresolved ref dir 791 index 0 namelen 25 name file filetype 1 errors 6, no dir index, no inode ref
        unresolved ref dir 3676 index 0 namelen 62 name file filetype 1 erro
rs 6, no dir index, no inode ref
ERROR: errors found in fs roots
WARNING: reserved space leaked, flag=0x2 bytes_reserved=32768
nt


deleting dev extent
deleting dev extent
deleting dev extent
deleting dev extent
deleting dev extent
reset devid 1 bytes_used to 1396938113024
No device size related problem found
cache and super generation don't match, space cache will be invalidated
found 43832565760 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 17694720
total fs tree bytes: 13598720
total extent tree bytes: 1212416
btree space waste bytes: 5081349
file data blocks allocated: 73458024448
 referenced 73454837760

After that check without repair still reports the same errors and mount still fails

[4035459.256276] BTRFS info (device dm-1): disk space caching is enabled
[4035459.276458] BTRFS info (device dm-1): has skinny extents
[4035459.357797] BTRFS error (device dm-1): chunk 20971520 has missing dev extent, have 0 expect 2
[4035459.385928] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117
[4035459.413170] BTRFS error (device dm-1): open_ctree failed

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-10 19:11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
@ 2022-06-10 19:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-06-11  0:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-06-13 17:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  0 siblings, 2 replies; 479+ messages in thread
From: Josef Bacik @ 2022-06-10 19:55 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Fri, Jun 10, 2022 at 3:11 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Fri, Jun 10, 2022 at 02:47:54PM -0400, Josef Bacik wrote:
> > Sorry I keep going back and forth on how to deal with this.  I've
> > pushed some code, you can run btrfs check --repair again and then try
> > again.  Thanks,
>
> Thanks, lots of output, not sure how much you want.
>
> For one:
> Device extent[1, 14803716276224, 1073741824] didn't find the relative chunk.
> Device extent[1, 14804790018048, 1073741824] didn't find the relative chunk.
> Device extent[1, 14805863759872, 1073741824] didn't find the relative chunk.
> Device extent[1, 14806937501696, 1073741824] didn't find the relative chunk.
> Device extent[1, 14808011243520, 1073741824] didn't find the relative chunk.
> super bytes used 21916233728 mismatches actual used 21916332032
> Block group[20971520, 8388608] (flags = 34) didn't find the relative chunk.
> Dev extent's total-byte(1396938113024) is not equal to byte-used(14599692746752) in dev[1, 216, 1]
> kernel-shared/extent-tree.c:2479: alloc_reserved_tree_block: Warning: assertion `1` failed, value 1
> ./btrfs(btrfs_run_delayed_refs+0x707)[0x5577947c927b]
> ./btrfs(btrfs_commit_transaction+0x3b)[0x5577947d7fb5]
> ./btrfs(repair_dev_item_bytes_used+0x6f)[0x5577948340d7]
> ./btrfs(+0x7d90b)[0x55779481690b]
> ./btrfs(+0x80193)[0x557794819193]
> ./btrfs(main+0x94)[0x5577947b1275]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f51c74847fd]
> ./btrfs(_start+0x2a)[0x5577947b0e1a]
> kernel-shared/extent-tree.c:2479: alloc_reserved_tree_block: Warning: assertion `1` failed, value 1
> ./btrfs(btrfs_run_delayed_refs+0x707)[0x5577947c927b]
> ./btrfs(btrfs_commit_transaction+0x3b)[0x5577947d7fb5]
> ./btrfs(repair_dev_item_bytes_used+0x6f)[0x5577948340d7]
> ./btrfs(+0x7d90b)[0x55779481690b]
> ./btrfs(+0x80193)[0x557794819193]
> ./btrfs(main+0x94)[0x5577947b1275]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f51c74847fd]
> ./btrfs(_start+0x2a)[0x5577947b0e1a]
> WARNING: reserved space leaked, transid=2608372 flag=0x2 bytes_reserved=32768
> [3/7] checking free space cache
> [4/7] checking fs roots
> root 161199 inode 54987 errors 1000, some csum missing
> WARNING: reserved space leaked, transid=2608373 flag=0x2 bytes_reserved=32768
> root 161199 inode 54988 errors 1100, file extent discount, some csum missing
> Found file extent holes:
>         start: 0, len: 1572864
> WARNING: reserved space leaked, transid=2608374 flag=0x2 bytes_reserved=32768
> root 161199 inode 54989 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong
> Found file extent holes:
>         start: 16252928, len: 91824128
>         unresolved ref dir 54974 index 16 namelen 18 name foo
>
> (...)
>
> Found file extent holes:
>         start: 197263360, len: 399777792
> root 164629 inode 73086 errors 1000, some csum missing
> WARNING: reserved space leaked, transid=2608425 flag=0x2 bytes_reserved=32768
> root 164629 inode 73094 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong
> Found file extent holes:
>         start: 0, len: 524288
>         unresolved ref dir 74963 index 32 namelen 56 name file filetype 0 errors 3
> , no dir item, no dir index
> WARNING: reserved space leaked, transid=2608426 flag=0x2 bytes_reserved=32768
> root 164629 inode 73097 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong
> Found file extent holes:
>         start: 0, len: 524288
>         unresolved ref dir 74963 index 36 namelen 56 name file filetype 0 errors 3
> , no dir item, no dir index
> root 164629 inode 73099 errors 1000, some csum missing
> root 164629 inode 73100 errors 1000, some csum missing
>         unresolved ref dir 791 index 0 namelen 25 name file filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 3676 index 0 namelen 62 name file filetype 1 erro
> rs 6, no dir index, no inode ref
> ERROR: errors found in fs roots
> WARNING: reserved space leaked, flag=0x2 bytes_reserved=32768
> nt
>
>
> deleting dev extent
> deleting dev extent
> deleting dev extent
> deleting dev extent
> deleting dev extent
> reset devid 1 bytes_used to 1396938113024
> No device size related problem found
> cache and super generation don't match, space cache will be invalidated
> found 43832565760 bytes used, error(s) found
> total csum bytes: 0
> total tree bytes: 17694720
> total fs tree bytes: 13598720
> total extent tree bytes: 1212416
> btree space waste bytes: 5081349
> file data blocks allocated: 73458024448
>  referenced 73454837760
>
> After that check without repair still reports the same errors and mount still fails
>
> [4035459.256276] BTRFS info (device dm-1): disk space caching is enabled
> [4035459.276458] BTRFS info (device dm-1): has skinny extents
> [4035459.357797] BTRFS error (device dm-1): chunk 20971520 has missing dev extent, have 0 expect 2
> [4035459.385928] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117
> [4035459.413170] BTRFS error (device dm-1): open_ctree failed

Gaahhh ok I see what I've been fucking up.  I had it in my head that
system chunks only exist in the super block, but they are both in the
super block so we can bootstrap the fs, _and_ in the chunk tree so we
don't confuse fsck.  This works out for the progs (and to a certain
extent the kernel) because it always has to read it from the super
block in order to mount, but it falls down in fsck (and in the kernel
like the above) when we don't find the same chunk in the chunk tree.

Soooooo I've fixed my idiocy and moved the code around.  Unfortunately
my last fix deleted the stripes from the sys array, so we need to get
those back.  So once again run

btrfs rescue recover-chunks <device>
btrfs rescue init-extent-tree <device>
btrfs check --repair <device>

and then it should be good to go as far as the chunks and the device
extents and everything.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-10 19:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
@ 2022-06-11  0:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-06-11 14:59                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-13 17:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-11  0:14 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Fri, Jun 10, 2022 at 03:55:09PM -0400, Josef Bacik wrote:
> Soooooo I've fixed my idiocy and moved the code around.  Unfortunately
> my last fix deleted the stripes from the sys array, so we need to get
> those back.  So once again run

Thanks for looking into this
 
> btrfs rescue recover-chunks <device>
> btrfs rescue init-extent-tree <device>
> btrfs check --repair <device>

Let's go for another round :)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
FS_INFO IS 0x559bd17b9bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x559bd17b9bc0
Walking all our trees and pinning down the currently accessible blocks
Found missing chunk in super block 20971520-29360128 type 34
adding bg for 20971520 8388608
kernel-shared/extent-tree.c:2829: btrfs_add_block_group: BUG_ON `ret` triggered, value -17
./btrfs(+0x29f27)[0x559bd01a8f27]
./btrfs(btrfs_add_block_group+0x1e0)[0x559bd01ad700]
./btrfs(btrfs_find_recover_chunks+0x4fe)[0x559bd020fc44]
./btrfs(+0x848ce)[0x559bd02038ce]
./btrfs(handle_command_group+0x49)[0x559bd019717b]
./btrfs(main+0x94)[0x559bd0197275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fe20f24a7fd]
./btrfs(_start+0x2a)[0x559bd0196e1a]
Aborted

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-11  0:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-06-11 14:59                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-12 17:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-12 17:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  0 siblings, 2 replies; 479+ messages in thread
From: Josef Bacik @ 2022-06-11 14:59 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Fri, Jun 10, 2022 at 8:14 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Fri, Jun 10, 2022 at 03:55:09PM -0400, Josef Bacik wrote:
> > Soooooo I've fixed my idiocy and moved the code around.  Unfortunately
> > my last fix deleted the stripes from the sys array, so we need to get
> > those back.  So once again run
>
> Thanks for looking into this
>
> > btrfs rescue recover-chunks <device>
> > btrfs rescue init-extent-tree <device>
> > btrfs check --repair <device>
>
> Let's go for another round :)
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> FS_INFO IS 0x559bd17b9bc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x559bd17b9bc0
> Walking all our trees and pinning down the currently accessible blocks
> Found missing chunk in super block 20971520-29360128 type 34
> adding bg for 20971520 8388608
> kernel-shared/extent-tree.c:2829: btrfs_add_block_group: BUG_ON `ret` triggered, value -17
> ./btrfs(+0x29f27)[0x559bd01a8f27]
> ./btrfs(btrfs_add_block_group+0x1e0)[0x559bd01ad700]
> ./btrfs(btrfs_find_recover_chunks+0x4fe)[0x559bd020fc44]
> ./btrfs(+0x848ce)[0x559bd02038ce]
> ./btrfs(handle_command_group+0x49)[0x559bd019717b]
> ./btrfs(main+0x94)[0x559bd0197275]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fe20f24a7fd]
> ./btrfs(_start+0x2a)[0x559bd0196e1a]
> Aborted

Oops, sorry about that, fixed it up.  My wife is travelling this week
so I'm going to be a little slower than normal, but hopefully we're
getting close to the end here.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-11 14:59                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-06-12 17:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-12 20:05                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-06-12 17:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-12 17:06 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sat, Jun 11, 2022 at 10:59:15AM -0400, Josef Bacik wrote:
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> > FS_INFO IS 0x559bd17b9bc0
> > Couldn't find the last root for 8
> > FS_INFO AFTER IS 0x559bd17b9bc0
> > Walking all our trees and pinning down the currently accessible blocks
> > Found missing chunk in super block 20971520-29360128 type 34
> > adding bg for 20971520 8388608
> > kernel-shared/extent-tree.c:2829: btrfs_add_block_group: BUG_ON `ret` triggered, value -17
> > ./btrfs(+0x29f27)[0x559bd01a8f27]
> > ./btrfs(btrfs_add_block_group+0x1e0)[0x559bd01ad700]
> > ./btrfs(btrfs_find_recover_chunks+0x4fe)[0x559bd020fc44]
> > ./btrfs(+0x848ce)[0x559bd02038ce]
> > ./btrfs(handle_command_group+0x49)[0x559bd019717b]
> > ./btrfs(main+0x94)[0x559bd0197275]
> > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fe20f24a7fd]
> > ./btrfs(_start+0x2a)[0x559bd0196e1a]
> > Aborted
> 
> Oops, sorry about that, fixed it up.  My wife is travelling this week
> so I'm going to be a little slower than normal, but hopefully we're
> getting close to the end here.  Thanks,

No worries. Same?

gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -5
commit affc3bcd741ca350bbf340e757698281b149c9f3
Author: Josef Bacik <josef@toxicpanda.com>
Date:   Sat Jun 11 10:58:21 2022 -0400

    deal with existing block groups
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
FS_INFO IS 0x561aa58a4bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x561aa58a4bc0
Walking all our trees and pinning down the currently accessible blocks
Found missing chunk in super block 20971520-29360128 type 34
adding bg for 20971520 8388608
kernel-shared/extent-tree.c:2829: btrfs_add_block_group: BUG_ON `ret` triggered, value -17
./btrfs(+0x29f27)[0x561aa569ef27]
./btrfs(btrfs_add_block_group+0x1e0)[0x561aa56a3700]
./btrfs(btrfs_find_recover_chunks+0x4fe)[0x561aa5705c44]
./btrfs(+0x848ce)[0x561aa56f98ce]
./btrfs(handle_command_group+0x49)[0x561aa568d17b]
./btrfs(main+0x94)[0x561aa568d275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f282855a7fd]
./btrfs(_start+0x2a)[0x561aa568ce1a]
Aborted
gargamel:/var/local/src/btrfs-progs-josefbacik# 
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-11 14:59                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-12 17:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-12 17:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-12 20:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-12 17:37 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sat, Jun 11, 2022 at 10:59:15AM -0400, Josef Bacik wrote:
> Oops, sorry about that, fixed it up.  My wife is travelling this week
> so I'm going to be a little slower than normal, but hopefully we're
> getting close to the end here.  Thanks,

In the meantime, I'm building the new replacement filesystem. I used
your tree to have the latest mkfs.btrfs.

gargamel:/var/local/src/btrfs-progs-josefbacik# ./mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1n
btrfs-progs v5.16.2 
See http://btrfs.wiki.kernel.org for more information.

NOTE: several default settings have changed in version 5.15, please make sure
      this does not affect your deployments:
      - DUP for metadata (-m dup)
      - enabled no-holes (-O no-holes)
      - enabled free-space-tree (-R free-space-tree)

Label:              dshelf1
UUID:               ee91f407-39cb-41ef-bd7b-89eee4504ad5
Node size:          16384
Sector size:        4096
Filesystem size:    43.66TiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, skinny-metadata, no-holes
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1    43.66TiB  /dev/mapper/dshelf1n

Does this look ok?
(it's still running on top of cryptsetup, which is on top of mdadm raid5, but this time
I added --consistency-policy=ppl and I removed bcache in the middle.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-12 17:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-12 20:05                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-06-12 21:19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-12 20:05 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Jun 12, 2022 at 1:06 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sat, Jun 11, 2022 at 10:59:15AM -0400, Josef Bacik wrote:
> > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> > > FS_INFO IS 0x559bd17b9bc0
> > > Couldn't find the last root for 8
> > > FS_INFO AFTER IS 0x559bd17b9bc0
> > > Walking all our trees and pinning down the currently accessible blocks
> > > Found missing chunk in super block 20971520-29360128 type 34
> > > adding bg for 20971520 8388608
> > > kernel-shared/extent-tree.c:2829: btrfs_add_block_group: BUG_ON `ret` triggered, value -17
> > > ./btrfs(+0x29f27)[0x559bd01a8f27]
> > > ./btrfs(btrfs_add_block_group+0x1e0)[0x559bd01ad700]
> > > ./btrfs(btrfs_find_recover_chunks+0x4fe)[0x559bd020fc44]
> > > ./btrfs(+0x848ce)[0x559bd02038ce]
> > > ./btrfs(handle_command_group+0x49)[0x559bd019717b]
> > > ./btrfs(main+0x94)[0x559bd0197275]
> > > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fe20f24a7fd]
> > > ./btrfs(_start+0x2a)[0x559bd0196e1a]
> > > Aborted
> >
> > Oops, sorry about that, fixed it up.  My wife is travelling this week
> > so I'm going to be a little slower than normal, but hopefully we're
> > getting close to the end here.  Thanks,
>
> No worries. Same?
>

Yeah I missed we were BUG_ON(ret), so my change does the right thing
if we get a failure, but I needed to update the core code to return
the error.  Should work this time, thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-12 17:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-12 20:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-06-12 21:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-12 20:06 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Jun 12, 2022 at 1:37 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sat, Jun 11, 2022 at 10:59:15AM -0400, Josef Bacik wrote:
> > Oops, sorry about that, fixed it up.  My wife is travelling this week
> > so I'm going to be a little slower than normal, but hopefully we're
> > getting close to the end here.  Thanks,
>
> In the meantime, I'm building the new replacement filesystem. I used
> your tree to have the latest mkfs.btrfs.
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1n
> btrfs-progs v5.16.2
> See http://btrfs.wiki.kernel.org for more information.
>
> NOTE: several default settings have changed in version 5.15, please make sure
>       this does not affect your deployments:
>       - DUP for metadata (-m dup)
>       - enabled no-holes (-O no-holes)
>       - enabled free-space-tree (-R free-space-tree)
>
> Label:              dshelf1
> UUID:               ee91f407-39cb-41ef-bd7b-89eee4504ad5
> Node size:          16384
> Sector size:        4096
> Filesystem size:    43.66TiB
> Block group profiles:
>   Data:             single            8.00MiB
>   Metadata:         DUP               1.00GiB
>   System:           DUP               8.00MiB
> SSD detected:       no
> Zoned device:       no
> Incompat features:  extref, skinny-metadata, no-holes
> Runtime features:   free-space-tree
> Checksum:           crc32c
> Number of devices:  1
> Devices:
>    ID        SIZE  PATH
>     1    43.66TiB  /dev/mapper/dshelf1n
>
> Does this look ok?

Yup this is everything I want to see right now.  Here's hoping the new
setup lasts more than a couple of years between catastrophes.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-12 20:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-06-12 21:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-06-12 21:14 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Jun 12, 2022 at 04:06:33PM -0400, Josef Bacik wrote:
> > gargamel:/var/local/src/btrfs-progs-josefbacik# ./mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1n
> > btrfs-progs v5.16.2
> > See http://btrfs.wiki.kernel.org for more information.
> >
> > NOTE: several default settings have changed in version 5.15, please make sure
> >       this does not affect your deployments:
> >       - DUP for metadata (-m dup)
> >       - enabled no-holes (-O no-holes)
> >       - enabled free-space-tree (-R free-space-tree)
> >
> > Label:              dshelf1
> > UUID:               ee91f407-39cb-41ef-bd7b-89eee4504ad5
> > Node size:          16384
> > Sector size:        4096
> > Filesystem size:    43.66TiB
> > Block group profiles:
> >   Data:             single            8.00MiB
> >   Metadata:         DUP               1.00GiB
> >   System:           DUP               8.00MiB
> > SSD detected:       no
> > Zoned device:       no
> > Incompat features:  extref, skinny-metadata, no-holes
> > Runtime features:   free-space-tree
> > Checksum:           crc32c
> > Number of devices:  1
> > Devices:
> >    ID        SIZE  PATH
> >     1    43.66TiB  /dev/mapper/dshelf1n
> >
> > Does this look ok?
> 
> Yup this is everything I want to see right now.  Here's hoping the new
> setup lasts more than a couple of years between catastrophes.  Thanks,

Thanks for confirming that the mkfs options look optimal.

I'm making 3 changes
1) no more bcache
2) raid5 write hole protection in mdadm (didn't have it before)
3) I'll make damn sure write caching is off on all the drives

Between those 3, it should be *much* safer, even if it will be slower
too.
Your work on fixing the FS is not for naught though, getting back a
damaged filesystem in some mountable state will be a great improvement
for btrfs and put it more on par with ext4 on recoverability.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-12 20:05                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-06-12 21:19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-06-12 22:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-12 21:19 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Sun, Jun 12, 2022 at 04:05:22PM -0400, Josef Bacik wrote:
> Yeah I missed we were BUG_ON(ret), so my change does the right thing
> if we get a failure, but I needed to update the core code to return
> the error.  Should work this time, thanks,


(gdb) run rescue recover-chunks /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue recover-chunks /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x555555652bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555652bc0
Walking all our trees and pinning down the currently accessible blocks
Found missing chunk in super block 20971520-29360128 type 34
adding bg for 20971520 8388608

Program received signal SIGSEGV, Segmentation fault.
setup_free_space (fs_info=0x555555652bc0) at cmds/rescue-recover-chunks.c:500
500                     bg->cached = 1;
(gdb) bt
#0  setup_free_space (fs_info=0x555555652bc0) at cmds/rescue-recover-chunks.c:500
#1  btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-recover-chunks.c:581
#2  0x00005555555d87d5 in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65
#3  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555648d40 <cmd_struct_rescue_recover_chunks>) at cmds/commands.h:125
#4  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#5  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555649cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#6  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-12 21:19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-06-12 22:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 0 replies; 479+ messages in thread
From: Josef Bacik @ 2022-06-12 22:32 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Sun, Jun 12, 2022 at 5:19 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Sun, Jun 12, 2022 at 04:05:22PM -0400, Josef Bacik wrote:
> > Yeah I missed we were BUG_ON(ret), so my change does the right thing
> > if we get a failure, but I needed to update the core code to return
> > the error.  Should work this time, thanks,
>
>
> (gdb) run rescue recover-chunks /dev/mapper/dshelf1
> Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue recover-chunks /dev/mapper/dshelf1
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> FS_INFO IS 0x555555652bc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x555555652bc0
> Walking all our trees and pinning down the currently accessible blocks
> Found missing chunk in super block 20971520-29360128 type 34
> adding bg for 20971520 8388608
>
> Program received signal SIGSEGV, Segmentation fault.
> setup_free_space (fs_info=0x555555652bc0) at cmds/rescue-recover-chunks.c:500
> 500                     bg->cached = 1;

Lord 3 stupid things in a row, maybe now we're good?  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-10 19:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
  2022-06-11  0:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-06-13 17:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-06-13 18:28                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  2022-06-13 18:29                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  1 sibling, 2 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-06-13 17:56 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

 #####                                                    ###
#     #  #    #   ####    ####   ######   ####    ####    ###
#        #    #  #    #  #    #  #       #       #        ###
 #####   #    #  #       #       #####    ####    ####     #
      #  #    #  #       #       #            #       #
#     #  #    #  #    #  #    #  #       #    #  #    #   ###
 #####    ####    ####    ####   ######   ####    ####    ###

On Fri, Jun 10, 2022 at 03:55:09PM -0400, Josef Bacik wrote:
> btrfs rescue recover-chunks <device>

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
FS_INFO IS 0x56528f943bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x56528f943bc0
Walking all our trees and pinning down the currently accessible blocks
No missing chunks, we're all done
doing close???
Recover chunks succeeded, you can run check now

> btrfs rescue init-extent-tree <device>

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x55fd039e2bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55fd039e2bc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups
deleting space cache for 20971520
deleting space cache for 11106814787584
deleting space cache for 11108962271232
deleting space cache for 11110036013056
deleting space cache for 11111109754880
(...)
doing roots
(...)
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 164633
processed 16384 of 75694080 possible bytes, 0%
Recording extents for root 164823
processed 507904 of 63504384 possible bytes, 0%
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes, 100%
doing block accounting
doing close???
Init extent tree finished, you can run check now

> btrfs check --repair <device>

FS_INFO AFTER IS 0x5641f2a63bd0
[1/7] checking root items
checksum verify failed on 15645959372800 wanted 0x847c08bf found 0x17a9e2f1
checksum verify failed on 15645959389184 wanted 0x3cc757a7 found 0x3b4eff03
checksum verify failed on 15645681451008 wanted 0x7516a3d9 found 0x97f7437d
checksum verify failed on 15646003970048 wanted 0xf18cc579 found 0x1bc64584
checksum verify failed on 15645867720704 wanted 0x14cc427a found 0x9f516106
checksum verify failed on 15645529604096 wanted 0xd11e24d5 found 0x8d01bc00
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 15645959356416 wanted 0x2fa8537e found 0x90ac1f4e
checksum verify failed on 15645692067840 wanted 0x7874ded3 found 0x1e94afcd
checksum verify failed on 15645529620480 wanted 0x9ba9c3df found 0x1813c193
checksum verify failed on 15645608165376 wanted 0x2af09d83 found 0xdc3aa13d
checksum verify failed on 15645815291904 wanted 0x27e465d0 found 0x3e898f04
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 15645815357440 wanted 0xeff7f183 found 0x21b9d056
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Fixed 0 roots.
[2/7] checking extents
Chunk[256, 228, 20971520] stripe[1, 20971520] is not found in dev extent
Chunk[256, 228, 20971520] stripe[1, 29360128] is not found in dev extent
[3/7] checking free space cache
(...)
root 164629 inode 73099 errors 1000, some csum missing
root 164629 inode 73100 errors 1000, some csum missing
        unresolved ref dir 791 index 0 namelen 25 name file filetype 1 errors 6, no dir index, no inode ref
        unresolved ref dir 3676 index 0 namelen 62 name file2 filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs roots

Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
No device size related problem found
cache and super generation don't match, space cache will be invalidated
found 21916200960 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8765440
total fs tree bytes: 6799360
total extent tree bytes: 573440
btree space waste bytes: 2514656
file data blocks allocated: 


gargamel:~# mount /dev/mapper/dshelf1 /mnt/mnt
gargamel:~#
[4289823.922324] BTRFS info (device dm-1): trying to use backup root at mount time
[4289823.922326] BTRFS info (device dm-1): disk space caching is enabled
[4289823.922327] BTRFS info (device dm-1): has skinny extents
[4289824.188614] BTRFS info (device dm-1): enabling ssd optimizations
[4289847.574926] BTRFS info (device dm-1): flagging fs with big metadata feature
[4289847.598104] BTRFS info (device dm-1): disk space caching is enabled
[4289847.651582] BTRFS info (device dm-1): has skinny extents
[4289847.699541] BTRFS info (device dm-1): enabling ssd optimizations
[4289847.730931] BTRFS info (device dm-1): checking UUID tree
[4289847.798826] BTRFS error (device dm-1): bad tree block level 19 on 15645959372800
[4289847.912586] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959372800 (dev /dev/mapper/dshelf1 sector 29403983072)
[4289847.956640] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959376896 (dev /dev/mapper/dshelf1 sector 29403983080)
[4289848.000894] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959380992 (dev /dev/mapper/dshelf1 sector 29403983088)
[4289848.045141] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959385088 (dev /dev/mapper/dshelf1 sector 29403983096)
[4289848.083771] BTRFS error (device dm-1): bad tree block level 39 on 15645959389184
[4289848.111468] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959389184 (dev /dev/mapper/dshelf1 sector 29403983104)
[4289848.155838] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959393280 (dev /dev/mapper/dshelf1 sector 29403983112)
[4289848.199940] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959397376 (dev /dev/mapper/dshelf1 sector 29403983120)
[4289848.244198] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959401472 (dev /dev/mapper/dshelf1 sector 29403983128)
[4289848.281912] BTRFS error (device dm-1): bad tree block level 85 on 15645681451008
[4289848.358339] BTRFS info (device dm-1): read error corrected: ino 0 off 15645681451008 (dev /dev/mapper/dshelf1 sector 29403440256)
[4289848.396605] BTRFS info (device dm-1): read error corrected: ino 0 off 15645681455104 (dev /dev/mapper/dshelf1 sector 29403440264)
[4289848.436740] BTRFS error (device dm-1): bad tree block level 22 on 15646003970048
[4289848.493665] BTRFS error (device dm-1): bad tree block level 127 on 15645867720704
[4289848.549485] BTRFS error (device dm-1): bad tree block level 165 on 15645529604096
[4289848.640033] BTRFS error (device dm-1): bad tree block level 32 on 15645781344256
[4289848.713594] BTRFS error (device dm-1): bad tree block level 16 on 15645959356416
[4289848.786591] BTRFS warning (device dm-1): checksum verify failed on 15645692067840 wanted 0x7874ded3 found 0x1e94afcd level 7
[4289848.837010] BTRFS error (device dm-1): bad tree block level 62 on 15645529620480
[4289848.905220] BTRFS error (device dm-1): bad tree block level 151 on 15645608165376
[4289848.941769] BTRFS error (device dm-1): bad tree block level 24 on 15645815291904
[4289848.991611] BTRFS error (device dm-1): bad tree block level 34 on 15645419667456
[4289849.061467] BTRFS error (device dm-1): bad tree block level 26 on 15645815357440
[4289849.109509] BTRFS error (device dm-1): bad tree block level 18 on 15645781196800

There is still some damage that maybe check/repair should fix, but it's
mountable, that's definitely a huge success!

Thanks Josef, that was a lot of work and determination :)

Let me know if there is more you'd like to look at, and/or try and get
the FS to a state where it's actually clean, but honestly as long as it
mounts, that's already a lot, obviously.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-13 17:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
@ 2022-06-13 18:28                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
  2022-06-13 18:29                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  1 sibling, 0 replies; 479+ messages in thread
From: Marc MERLIN @ 2022-06-13 18:28 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

Ok, yes and no, it mounted, but 99% of the files are missing.
There are a few in lost+found but ultimately almost everything
is gone.

The top structure is there, but then mostly empty underneath:

drwxr-xr-x 1 root   root              0 Jun 13 11:26 .beeshome/
-rw-r--r-- 1 root   root    13569961558 Jul 16  2020 DS1
drwx------ 1 root   root           5062 Jun  8 15:52 lost+found/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media/
d????????? ? ?      ?                 ?            ? media_daily.20220325_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_daily.20220326_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_daily.20220327_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_daily.20220328_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_hourly.20220328_22:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_hourly.20220328_23:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_hourly.20220329_00:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_hourly.20220329_01:01:01/
drwxr-xr-x 1 root   root             28 Jun  8 16:12 media_ro.20220227_00:24:39/
drwxr-xr-x 1 root   root             28 Jun  8 16:22 media_ro.20220306_00:24:43/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_ro.20220313_00:24:44/
drwxr-xr-x 1 root   root             28 Jun  8 16:42 media_ro.20220320_00:24:47/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_ro.20220327_00:24:32/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_weekly.20220130_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_weekly.20220206_02:03:01/
drwxr-xr-x 1 root   root             20 Jun  8 16:00 media_weekly.20220213_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_weekly.20220306_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other/
d????????? ? ?      ?                 ?            ? other_daily.20220325_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_daily.20220326_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_daily.20220327_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_daily.20220328_01:02:01/
d????????? ? ?      ?                 ?            ? other_hourly.20220328_21:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_hourly.20220328_22:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_hourly.20220328_23:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_hourly.20220329_00:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_hourly.20220329_01:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_ro.20220227_00:25:38/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_ro.20220306_00:25:40/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_ro.20220313_00:26:11/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_ro.20220320_00:25:35/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_ro.20220327_00:24:58/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_weekly.20220130_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_weekly.20220206_02:03:01/
drwxr-xr-x 1 root   root             28 Jun  8 16:06 other_weekly.20220213_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_weekly.20220306_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 SW/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 SW_ro.20220227_00:34:48/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 SW_ro.20220306_00:33:55/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 SW_ro.20220313_00:33:42/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 SW_ro.20220320_00:35:42/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 SW_ro.20220327_00:32:46/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound/
d????????? ? ?      ?                 ?            ? Sound_daily.20220325_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_daily.20220326_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_daily.20220327_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_daily.20220328_01:02:01/
d????????? ? ?      ?                 ?            ? Sound_hourly.20220328_22:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_hourly.20220328_23:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_hourly.20220329_00:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_hourly.20220329_01:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_ro.20220227_00:40:41/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_ro.20220306_00:45:46/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_ro.20220313_00:37:25/
drwxr-xr-x 1 merlin hideftp          70 Jun 25  2021 Sound_ro.20220320_00:39:38/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_ro.20220327_00:35:22/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_weekly.20220130_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_weekly.20220206_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_weekly.20220213_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_weekly.20220306_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2/
d????????? ? ?      ?                 ?            ? Med2_daily.20220325_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_daily.20220326_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_daily.20220327_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_daily.20220328_01:02:01/
d????????? ? ?      ?                 ?            ? Med2_hourly.20220328_22:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_hourly.20220328_23:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_hourly.20220329_00:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_hourly.20220329_01:01:01/
drwxr-xr-x 1 root   root             20 Jun  8 16:15 Med2_ro.20220227_00:46:32/
drwxr-xr-x 1 root   root             20 Jun  8 16:24 Med2_ro.20220306_00:46:14/
drwxr-xr-x 1 root   root             20 Jun  8 16:42 Med2_ro.20220313_00:38:01/
drwxr-xr-x 1 merlin hideftp         102 Jun 25  2021 Med2_ro.20220320_00:40:27/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_ro.20220327_00:35:50/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_weekly.20220130_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_weekly.20220206_02:03:01/
drwxr-xr-x 1 root   root             20 Jun  8 15:54 Med2_weekly.20220213_02:03:01/
drwxr-xr-x 1 root   root             20 Jun  8 16:35 Med2_weekly.20220306_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 W/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 W_ro.20220227_02:10:25/
drwxr-xr-x 1 root   root           4498 Mar 28 11:01 W_ro.20220306_00:47:32/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 W_ro.20220313_00:38:28/
drwxr-xr-x 1 root   root             34 Jun  8 17:52 W_ro.20220320_01:12:38/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 W_ro.20220327_00:37:15/


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-13 17:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
  2022-06-13 18:28                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
@ 2022-06-13 18:29                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-13 20:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-15 14:29                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  1 sibling, 2 replies; 479+ messages in thread
From: Josef Bacik @ 2022-06-13 18:29 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Jun 13, 2022 at 1:56 PM Marc MERLIN <marc@merlins.org> wrote:
>
>  #####                                                    ###
> #     #  #    #   ####    ####   ######   ####    ####    ###
> #        #    #  #    #  #    #  #       #       #        ###
>  #####   #    #  #       #       #####    ####    ####     #
>       #  #    #  #       #       #            #       #
> #     #  #    #  #    #  #    #  #       #    #  #    #   ###
>  #####    ####    ####    ####   ######   ####    ####    ###
>
> On Fri, Jun 10, 2022 at 03:55:09PM -0400, Josef Bacik wrote:
> > btrfs rescue recover-chunks <device>
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
> FS_INFO IS 0x56528f943bc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x56528f943bc0
> Walking all our trees and pinning down the currently accessible blocks
> No missing chunks, we're all done
> doing close???
> Recover chunks succeeded, you can run check now
>
> > btrfs rescue init-extent-tree <device>
>
> gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
> FS_INFO IS 0x55fd039e2bc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x55fd039e2bc0
> Walking all our trees and pinning down the currently accessible blocks
> Clearing the extent root and re-init'ing the block groups
> deleting space cache for 20971520
> deleting space cache for 11106814787584
> deleting space cache for 11108962271232
> deleting space cache for 11110036013056
> deleting space cache for 11111109754880
> (...)
> doing roots
> (...)
> processed 49152 of 49463296 possible bytes, 0%
> Recording extents for root 164633
> processed 16384 of 75694080 possible bytes, 0%
> Recording extents for root 164823
> processed 507904 of 63504384 possible bytes, 0%
> Recording extents for root 18446744073709551607
> processed 16384 of 16384 possible bytes, 100%
> doing block accounting
> doing close???
> Init extent tree finished, you can run check now
>
> > btrfs check --repair <device>
>
> FS_INFO AFTER IS 0x5641f2a63bd0
> [1/7] checking root items
> checksum verify failed on 15645959372800 wanted 0x847c08bf found 0x17a9e2f1
> checksum verify failed on 15645959389184 wanted 0x3cc757a7 found 0x3b4eff03
> checksum verify failed on 15645681451008 wanted 0x7516a3d9 found 0x97f7437d
> checksum verify failed on 15646003970048 wanted 0xf18cc579 found 0x1bc64584
> checksum verify failed on 15645867720704 wanted 0x14cc427a found 0x9f516106
> checksum verify failed on 15645529604096 wanted 0xd11e24d5 found 0x8d01bc00
> checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
> checksum verify failed on 15645959356416 wanted 0x2fa8537e found 0x90ac1f4e
> checksum verify failed on 15645692067840 wanted 0x7874ded3 found 0x1e94afcd
> checksum verify failed on 15645529620480 wanted 0x9ba9c3df found 0x1813c193
> checksum verify failed on 15645608165376 wanted 0x2af09d83 found 0xdc3aa13d
> checksum verify failed on 15645815291904 wanted 0x27e465d0 found 0x3e898f04
> checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
> checksum verify failed on 15645815357440 wanted 0xeff7f183 found 0x21b9d056
> checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
> Fixed 0 roots.
> [2/7] checking extents
> Chunk[256, 228, 20971520] stripe[1, 20971520] is not found in dev extent
> Chunk[256, 228, 20971520] stripe[1, 29360128] is not found in dev extent
> [3/7] checking free space cache
> (...)
> root 164629 inode 73099 errors 1000, some csum missing
> root 164629 inode 73100 errors 1000, some csum missing
>         unresolved ref dir 791 index 0 namelen 25 name file filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 3676 index 0 namelen 62 name file2 filetype 1 errors 6, no dir index, no inode ref
> ERROR: errors found in fs roots
>
> Starting repair.
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/dshelf1
> UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
> No device size related problem found
> cache and super generation don't match, space cache will be invalidated
> found 21916200960 bytes used, error(s) found
> total csum bytes: 0
> total tree bytes: 8765440
> total fs tree bytes: 6799360
> total extent tree bytes: 573440
> btree space waste bytes: 2514656
> file data blocks allocated:
>
>
> gargamel:~# mount /dev/mapper/dshelf1 /mnt/mnt
> gargamel:~#
> [4289823.922324] BTRFS info (device dm-1): trying to use backup root at mount time
> [4289823.922326] BTRFS info (device dm-1): disk space caching is enabled
> [4289823.922327] BTRFS info (device dm-1): has skinny extents
> [4289824.188614] BTRFS info (device dm-1): enabling ssd optimizations
> [4289847.574926] BTRFS info (device dm-1): flagging fs with big metadata feature
> [4289847.598104] BTRFS info (device dm-1): disk space caching is enabled
> [4289847.651582] BTRFS info (device dm-1): has skinny extents
> [4289847.699541] BTRFS info (device dm-1): enabling ssd optimizations
> [4289847.730931] BTRFS info (device dm-1): checking UUID tree
> [4289847.798826] BTRFS error (device dm-1): bad tree block level 19 on 15645959372800
> [4289847.912586] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959372800 (dev /dev/mapper/dshelf1 sector 29403983072)
> [4289847.956640] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959376896 (dev /dev/mapper/dshelf1 sector 29403983080)
> [4289848.000894] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959380992 (dev /dev/mapper/dshelf1 sector 29403983088)
> [4289848.045141] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959385088 (dev /dev/mapper/dshelf1 sector 29403983096)
> [4289848.083771] BTRFS error (device dm-1): bad tree block level 39 on 15645959389184
> [4289848.111468] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959389184 (dev /dev/mapper/dshelf1 sector 29403983104)
> [4289848.155838] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959393280 (dev /dev/mapper/dshelf1 sector 29403983112)
> [4289848.199940] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959397376 (dev /dev/mapper/dshelf1 sector 29403983120)
> [4289848.244198] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959401472 (dev /dev/mapper/dshelf1 sector 29403983128)
> [4289848.281912] BTRFS error (device dm-1): bad tree block level 85 on 15645681451008
> [4289848.358339] BTRFS info (device dm-1): read error corrected: ino 0 off 15645681451008 (dev /dev/mapper/dshelf1 sector 29403440256)
> [4289848.396605] BTRFS info (device dm-1): read error corrected: ino 0 off 15645681455104 (dev /dev/mapper/dshelf1 sector 29403440264)
> [4289848.436740] BTRFS error (device dm-1): bad tree block level 22 on 15646003970048
> [4289848.493665] BTRFS error (device dm-1): bad tree block level 127 on 15645867720704
> [4289848.549485] BTRFS error (device dm-1): bad tree block level 165 on 15645529604096
> [4289848.640033] BTRFS error (device dm-1): bad tree block level 32 on 15645781344256
> [4289848.713594] BTRFS error (device dm-1): bad tree block level 16 on 15645959356416
> [4289848.786591] BTRFS warning (device dm-1): checksum verify failed on 15645692067840 wanted 0x7874ded3 found 0x1e94afcd level 7
> [4289848.837010] BTRFS error (device dm-1): bad tree block level 62 on 15645529620480
> [4289848.905220] BTRFS error (device dm-1): bad tree block level 151 on 15645608165376
> [4289848.941769] BTRFS error (device dm-1): bad tree block level 24 on 15645815291904
> [4289848.991611] BTRFS error (device dm-1): bad tree block level 34 on 15645419667456
> [4289849.061467] BTRFS error (device dm-1): bad tree block level 26 on 15645815357440
> [4289849.109509] BTRFS error (device dm-1): bad tree block level 18 on 15645781196800
>
> There is still some damage that maybe check/repair should fix, but it's
> mountable, that's definitely a huge success!
>
> Thanks Josef, that was a lot of work and determination :)
>
> Let me know if there is more you'd like to look at, and/or try and get
> the FS to a state where it's actually clean, but honestly as long as it
> mounts, that's already a lot, obviously.
>

Alright so we have the missing csum things, which can be fixed with

btrfs rescue init-csum-tree <device>

now that everything is back the way it's supposed to be.

After that you can mount the fs and run a scrub, because you have
copies of metadata that are fucked, but copies that were ok, which is
why I disabled the block error warnings since they were confusing me.
Scrub should go along and fix all of that up.

The next thing is to fix the fs errors, which I imagine will cause
other problems like not being able to find directories and files and
such.  Once we've gotten the csum tree and the scrub done we can
tackle the remaining fs error problems which should be less terrifying
to mess with.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-13 18:29                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
@ 2022-06-13 20:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-13 22:19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-06-15 14:29                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-13 20:46 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Jun 13, 2022 at 02:29:30PM -0400, Josef Bacik wrote:
> Alright so we have the missing csum things, which can be fixed with
> 
> btrfs rescue init-csum-tree <device>

(gdb) run rescue init-csum-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-csum-tree /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x555555652bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555652bc0
processed 0 of 21907435520 possible data bytes, 0%
Program received signal SIGSEGV, Segmentation fault.
0x00005555555e351a in record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762
1762    BTRFS_SETGET_FUNCS(inode_flags, struct btrfs_inode_item, flags, 64);
(gdb) bt
#0  0x00005555555e351a in record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762
#1  0x00005555555e3906 in record_csums (root=root@entry=0x5555556f15d0, processed=processed@entry=0x7fffffffdbe8) at cmds/rescue-init-csum-tree.c:292
#2  0x00005555555e3f0f in foreach_root (cb=0x5555555e38fa <record_csums>, fs_info=0x555555652bc0) at cmds/rescue-init-csum-tree.c:338
#3  btrfs_init_csum_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-init-csum-tree.c:408
#4  0x00005555555d88b1 in cmd_rescue_init_csum_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:102
#5  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555648ce0 <cmd_struct_rescue_init_csum_tree>) at cmds/commands.h:125
#6  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555649cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-13 20:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-13 22:19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
  2022-06-13 23:52                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-13 22:19 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Jun 13, 2022 at 4:46 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Jun 13, 2022 at 02:29:30PM -0400, Josef Bacik wrote:
> > Alright so we have the missing csum things, which can be fixed with
> >
> > btrfs rescue init-csum-tree <device>
>
> (gdb) run rescue init-csum-tree /dev/mapper/dshelf1
> Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-csum-tree /dev/mapper/dshelf1
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> FS_INFO IS 0x555555652bc0
> Couldn't find the last root for 8
> FS_INFO AFTER IS 0x555555652bc0
> processed 0 of 21907435520 possible data bytes, 0%
> Program received signal SIGSEGV, Segmentation fault.
> 0x00005555555e351a in record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762
> 1762    BTRFS_SETGET_FUNCS(inode_flags, struct btrfs_inode_item, flags, 64);
> (gdb) bt
> #0  0x00005555555e351a in record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762
> #1  0x00005555555e3906 in record_csums (root=root@entry=0x5555556f15d0, processed=processed@entry=0x7fffffffdbe8) at cmds/rescue-init-csum-tree.c:292
> #2  0x00005555555e3f0f in foreach_root (cb=0x5555555e38fa <record_csums>, fs_info=0x555555652bc0) at cmds/rescue-init-csum-tree.c:338
> #3  btrfs_init_csum_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-init-csum-tree.c:408
> #4  0x00005555555d88b1 in cmd_rescue_init_csum_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:102
> #5  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555648ce0 <cmd_struct_rescue_init_csum_tree>) at cmds/commands.h:125
> #6  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
> #7  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555649cc0 <cmd_struct_rescue>) at cmds/commands.h:125
> #8  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Hmm that's not good, I've pushed a patch to see if what I think is
happening is actually happening.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-13 22:19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
@ 2022-06-13 23:52                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
  2022-06-15  1:44                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-13 23:52 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Jun 13, 2022 at 06:19:05PM -0400, Josef Bacik wrote:
> Hmm that's not good, I've pushed a patch to see if what I think is
> happening is actually happening.  Thanks,

Same?

(gdb) run rescue init-csum-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-csum-tree /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x555555652bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555652bc0
processed 0 of 21907435520 possible data bytes, 0%
Program received signal SIGSEGV, Segmentation fault.
record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762
1762    BTRFS_SETGET_FUNCS(inode_flags, struct btrfs_inode_item, flags, 64);
(gdb) bt
#0  record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762
#1  0x00005555555e394b in record_csums (root=root@entry=0x5555556f15d0, processed=processed@entry=0x7fffffffdbe8) at cmds/rescue-init-csum-tree.c:299
#2  0x00005555555e3f54 in foreach_root (cb=0x5555555e393f <record_csums>, fs_info=0x555555652bc0) at cmds/rescue-init-csum-tree.c:345
#3  btrfs_init_csum_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-init-csum-tree.c:415
#4  0x00005555555d88b1 in cmd_rescue_init_csum_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:102
#5  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555648ce0 <cmd_struct_rescue_init_csum_tree>) at cmds/commands.h:125
#6  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555649cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
(gdb) quit
A debugging session is active.

        Inferior 1 [process 30169] will be killed.

Quit anyway? (y or n) y
gargamel:/var/local/src/btrfs-progs-josefbacik# git pull
Current branch for-marc is up to date.

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-13 23:52                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
@ 2022-06-15  1:44                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
  0 siblings, 0 replies; 479+ messages in thread
From: Josef Bacik @ 2022-06-15  1:44 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Mon, Jun 13, 2022 at 7:52 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Mon, Jun 13, 2022 at 06:19:05PM -0400, Josef Bacik wrote:
> > Hmm that's not good, I've pushed a patch to see if what I think is
> > happening is actually happening.  Thanks,
>
> Same?
>

Sorry Marc, saw this while putting kids to bed and immediately forgot
it.  I think I've spotted the problem, you should be good now.
Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-13 18:29                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
  2022-06-13 20:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-15 14:29                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
  2022-06-15 14:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  1 sibling, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-15 14:29 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Mon, Jun 13, 2022 at 02:29:30PM -0400, Josef Bacik wrote:
> Alright so we have the missing csum things, which can be fixed with
> 
> btrfs rescue init-csum-tree <device>

Took a while, but it worked now, thanks.
 
(gdb) run rescue init-csum-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-csum-tree /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x555555652bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555652bc0
processed 21904289792 of 21907435520 possible data bytes, 99%doing close???
Init csum tree finished, you can run check now
> 
> After that you can mount the fs and run a scrub, because you have
> copies of metadata that are fucked, but copies that were ok, which is
> why I disabled the block error warnings since they were confusing me.
> Scrub should go along and fix all of that up.

gargamel:/mnt/mnt# btrfs scrub start -B .
running that now, I expect it will take a while.

> The next thing is to fix the fs errors, which I imagine will cause
> other problems like not being able to find directories and files and
> such.  Once we've gotten the csum tree and the scrub done we can
> tackle the remaining fs error problems which should be less terrifying
> to mess with.  Thanks,

would that be check --repair ?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-15 14:29                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
@ 2022-06-15 14:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
  2022-06-15 21:18                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-15 14:55 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 15, 2022 at 07:29:29AM -0700, Marc MERLIN wrote:
> gargamel:/mnt/mnt# btrfs scrub start -B .
> running that now, I expect it will take a while.
 
Never mind, it was fast:
gargamel:/mnt/mnt# btrfs scrub start -B .
scrub done for 96539b8c-ccc9-47bf-9e6c-29305890941e
Scrub started:    Wed Jun 15 07:28:02 2022
Status:           finished
Duration:         0:03:33
Total to scrub:   111.00GiB
Rate:             98.39MiB/s
Error summary:    verify=14
  Corrected:      14
  Uncorrectable:  0
  Unverified:     0
WARNING: errors detected during scrubbing, corrected

> > The next thing is to fix the fs errors, which I imagine will cause
> > other problems like not being able to find directories and files and
> > such.  Once we've gotten the csum tree and the scrub done we can
> > tackle the remaining fs error problems which should be less terrifying
> > to mess with.  Thanks,
> 
> would that be check --repair ?

here's check without repair:

[1/7] checking root items
[2/7] checking extents
ref mismatch on [11160501911552 16384] extent item 1, found 0
backref 11160501911552 root 7 not referenced back 0x56389bce5cd0
incorrect global backref count on 11160501911552 found 1 wanted 0
backpointer mismatch on [11160501911552 16384]
owner ref check failed [11160501911552 16384]
ref mismatch on [11160502042624 16384] extent item 1, found 0
backref 11160502042624 root 7 not referenced back 0x56389bce44d0
incorrect global backref count on 11160502042624 found 1 wanted 0
backpointer mismatch on [11160502042624 16384]
owner ref check failed [11160502042624 16384]
ref mismatch on [11160502845440 16384] extent item 1, found 0
backref 11160502845440 root 7 not referenced back 0x56389be42060
incorrect global backref count on 11160502845440 found 1 wanted 0
backpointer mismatch on [11160502845440 16384]
owner ref check failed [11160502845440 16384]
ref mismatch on [11160502927360 16384] extent item 1, found 0
backref 11160502927360 root 7 not referenced back 0x56389be42560
incorrect global backref count on 11160502927360 found 1 wanted 0
backpointer mismatch on [11160502927360 16384]
owner ref check failed [11160502927360 16384]
ref mismatch on [15645021241344 16384] extent item 1, found 0
backref 15645021241344 root 7 not referenced back 0x56389bdc98a0
incorrect global backref count on 15645021241344 found 1 wanted 0
backpointer mismatch on [15645021241344 16384]
owner ref check failed [15645021241344 16384]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
Couldn't find free space inode 1
Couldn't find free space inode 1
(...)
Couldn't find free space inode 1
Couldn't find free space inode 1
[4/7] checking fs roots
root 161199 inode 54988 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 1572864
root 161199 inode 54989 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 16252928, len: 91824128
	unresolved ref dir 54974 index 16 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55003 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 1048576
root 161199 inode 55004 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 1048576
	unresolved ref dir 56235 index 12 name foo filetype mismatch
root 161199 inode 55409 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 524288
	unresolved ref dir 55399 index 11 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55410 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 1572864
root 161199 inode 55411 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 524288
	unresolved ref dir 55399 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55412 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 1048576
	unresolved ref dir 55399 index 14 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55413 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 24117248, len: 34611200
	unresolved ref dir 55399 index 15 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55459 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 56098816, len: 265449472
	unresolved ref dir 55437 index 23 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55468 errors 100, file extent discount
Found file extent holes:
	start: 27787264, len: 302174208
root 161199 inode 55475 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 1048576
root 161199 inode 55476 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 28803072, len: 459870208
	unresolved ref dir 55437 index 40 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55526 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 524288
root 161199 inode 55527 errors 100, file extent discount
Found file extent holes:
	start: 26738688, len: 462848
root 161199 inode 55528 errors 100, file extent discount
Found file extent holes:
	start: 16252928, len: 184320
root 161199 inode 55530 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 524288
root 161199 inode 55531 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 4194304, len: 11001856
	unresolved ref dir 55511 index 21 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 54988 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 1572864
root 161889 inode 54989 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 16252928, len: 91824128
	unresolved ref dir 54974 index 16 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55003 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 1048576
root 161889 inode 55004 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 1048576
	unresolved ref dir 56235 index 48 name foo filetype mismatch
root 161889 inode 55409 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 524288
	unresolved ref dir 55399 index 11 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55410 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 1572864
root 161889 inode 55411 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 524288
	unresolved ref dir 55399 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55412 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 1048576
	unresolved ref dir 55399 index 14 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55413 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 24117248, len: 34611200
	unresolved ref dir 55399 index 15 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55459 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 56098816, len: 265449472
	unresolved ref dir 55437 index 23 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55468 errors 100, file extent discount
Found file extent holes:
	start: 27787264, len: 302174208
root 161889 inode 55475 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 1048576
root 161889 inode 55476 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 28803072, len: 459870208
	unresolved ref dir 55437 index 40 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55526 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 524288
root 161889 inode 55527 errors 100, file extent discount
Found file extent holes:
	start: 26738688, len: 462848
root 161889 inode 55528 errors 100, file extent discount
Found file extent holes:
	start: 16252928, len: 184320
root 161889 inode 55530 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 524288
root 161889 inode 55531 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 4194304, len: 11001856
	unresolved ref dir 55511 index 21 name foo filetype 0 errors 3, no dir item, no dir index
root 163920 inode 76551 errors 100, file extent discount
Found file extent holes:
	start: 1037262848, len: 184238080
root 163920 inode 76556 errors 100, file extent discount
Found file extent holes:
	start: 1639038976, len: 213225472
	unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
	unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
	unresolved ref dir 72785 index 720 name foo filetype mismatch
	unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
	unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
	unresolved ref dir 73103 index 544 name foo filetype mismatch
	unresolved ref dir 73103 index 672 name foo filetype mismatch
	unresolved ref dir 4179 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
	unresolved ref dir 4698 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
	unresolved ref dir 5506 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
	unresolved ref dir 5546 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
root 164624 inode 25918 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 1262850048
root 164624 inode 72429 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 1333788672, len: 3217154048
	unresolved ref dir 72438 index 4 name foo filetype 0 errors 3, no dir item, no dir index
root 164624 inode 72433 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 892338176, len: 14957748224
	unresolved ref dir 34951 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 164624 inode 72593 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 42074112, len: 413642752
	unresolved ref dir 75036 index 19 name foo filetype 0 errors 3, no dir item, no dir index
	unresolved ref dir 73103 index 1524 name foo filetype mismatch
root 164624 inode 73009 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 13869056, len: 53100544
	unresolved ref dir 72672 index 134 name foo filetype 1 errors 1, no dir item
root 164624 inode 73083 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 197263360, len: 399777792
root 164624 inode 73094 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 524288
	unresolved ref dir 74963 index 32 name foo filetype 0 errors 3, no dir item, no dir index
root 164624 inode 73097 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 524288
	unresolved ref dir 74963 index 36 name foo filetype 0 errors 3, no dir item, no dir index
	unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
	unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
	unresolved ref dir 73103 index 540 name foo filetype mismatch
	unresolved ref dir 4179 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
	unresolved ref dir 4698 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
	unresolved ref dir 5506 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
	unresolved ref dir 5546 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 39921 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 759824384, len: 2143326208
	unresolved ref dir 10205 index 356 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72429 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 1333788672, len: 3217154048
	unresolved ref dir 72438 index 4 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72433 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 892338176, len: 14957748224
	unresolved ref dir 34951 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72593 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 42074112, len: 413642752
	unresolved ref dir 75036 index 19 name foo filetype 0 errors 3, no dir item, no dir index
	unresolved ref dir 73103 index 1386 name foo filetype mismatch
root 164629 inode 73009 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 13869056, len: 53100544
	unresolved ref dir 72672 index 134 name foo filetype 1 errors 1, no dir item
root 164629 inode 73083 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 197263360, len: 399777792
root 164629 inode 73094 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 524288
	unresolved ref dir 74963 index 32 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 73097 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 524288
	unresolved ref dir 74963 index 36 name foo filetype 0 errors 3, no dir item, no dir index
	unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
	unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs roots
Opening filesystem to check...
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
found 21941104640 bytes used, error(s) found
total csum bytes: 21390908
total tree bytes: 33587200
total fs tree bytes: 6799360
total extent tree bytes: 655360
btree space waste bytes: 5863130
file data blocks allocated: 36729012224
 referenced 36727418880

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-15 14:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
@ 2022-06-15 21:18                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
  2022-06-15 21:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-15 21:18 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 15, 2022 at 10:55 AM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Jun 15, 2022 at 07:29:29AM -0700, Marc MERLIN wrote:
> > gargamel:/mnt/mnt# btrfs scrub start -B .
> > running that now, I expect it will take a while.
>
> Never mind, it was fast:
> gargamel:/mnt/mnt# btrfs scrub start -B .
> scrub done for 96539b8c-ccc9-47bf-9e6c-29305890941e
> Scrub started:    Wed Jun 15 07:28:02 2022
> Status:           finished
> Duration:         0:03:33
> Total to scrub:   111.00GiB

Hrm shit, this isn't good, don't you have a lot more data than 111gib?

> Rate:             98.39MiB/s
> Error summary:    verify=14
>   Corrected:      14
>   Uncorrectable:  0
>   Unverified:     0
> WARNING: errors detected during scrubbing, corrected
>
> > > The next thing is to fix the fs errors, which I imagine will cause
> > > other problems like not being able to find directories and files and
> > > such.  Once we've gotten the csum tree and the scrub done we can
> > > tackle the remaining fs error problems which should be less terrifying
> > > to mess with.  Thanks,
> >
> > would that be check --repair ?
>
> here's check without repair:
>
> [1/7] checking root items
> [2/7] checking extents
> ref mismatch on [11160501911552 16384] extent item 1, found 0
> backref 11160501911552 root 7 not referenced back 0x56389bce5cd0
> incorrect global backref count on 11160501911552 found 1 wanted 0
> backpointer mismatch on [11160501911552 16384]
> owner ref check failed [11160501911552 16384]
> ref mismatch on [11160502042624 16384] extent item 1, found 0
> backref 11160502042624 root 7 not referenced back 0x56389bce44d0
> incorrect global backref count on 11160502042624 found 1 wanted 0
> backpointer mismatch on [11160502042624 16384]
> owner ref check failed [11160502042624 16384]
> ref mismatch on [11160502845440 16384] extent item 1, found 0
> backref 11160502845440 root 7 not referenced back 0x56389be42060
> incorrect global backref count on 11160502845440 found 1 wanted 0
> backpointer mismatch on [11160502845440 16384]
> owner ref check failed [11160502845440 16384]
> ref mismatch on [11160502927360 16384] extent item 1, found 0
> backref 11160502927360 root 7 not referenced back 0x56389be42560
> incorrect global backref count on 11160502927360 found 1 wanted 0
> backpointer mismatch on [11160502927360 16384]
> owner ref check failed [11160502927360 16384]
> ref mismatch on [15645021241344 16384] extent item 1, found 0
> backref 15645021241344 root 7 not referenced back 0x56389bdc98a0
> incorrect global backref count on 15645021241344 found 1 wanted 0
> backpointer mismatch on [15645021241344 16384]
> owner ref check failed [15645021241344 16384]

Oh oops, I must have missed this in the init-extent-tree.  Let me look
into this and I'll let you know when you can run the code again.

> ERROR: errors found in extent allocation tree or chunk allocation
> [3/7] checking free space cache
> Couldn't find free space inode 1
> Couldn't find free space inode 1
> (...)
> Couldn't find free space inode 1
> Couldn't find free space inode 1
> [4/7] checking fs roots
> root 161199 inode 54988 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 1572864
> root 161199 inode 54989 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 16252928, len: 91824128
>         unresolved ref dir 54974 index 16 name foo filetype 0 errors 3, no dir item, no dir index
> root 161199 inode 55003 errors 500, file extent discount, nbytes wrong
> Found file extent holes:
>         start: 0, len: 1048576
> root 161199 inode 55004 errors 500, file extent discount, nbytes wrong
> Found file extent holes:
>         start: 0, len: 1048576
>         unresolved ref dir 56235 index 12 name foo filetype mismatch
> root 161199 inode 55409 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 0, len: 524288
>         unresolved ref dir 55399 index 11 name foo filetype 0 errors 3, no dir item, no dir index
> root 161199 inode 55410 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 1572864
> root 161199 inode 55411 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 0, len: 524288
>         unresolved ref dir 55399 index 13 name foo filetype 0 errors 3, no dir item, no dir index
> root 161199 inode 55412 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 0, len: 1048576
>         unresolved ref dir 55399 index 14 name foo filetype 0 errors 3, no dir item, no dir index
> root 161199 inode 55413 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 24117248, len: 34611200
>         unresolved ref dir 55399 index 15 name foo filetype 0 errors 3, no dir item, no dir index
> root 161199 inode 55459 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 56098816, len: 265449472
>         unresolved ref dir 55437 index 23 name foo filetype 0 errors 3, no dir item, no dir index
> root 161199 inode 55468 errors 100, file extent discount
> Found file extent holes:
>         start: 27787264, len: 302174208
> root 161199 inode 55475 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 1048576
> root 161199 inode 55476 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 28803072, len: 459870208
>         unresolved ref dir 55437 index 40 name foo filetype 0 errors 3, no dir item, no dir index
> root 161199 inode 55526 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 524288
> root 161199 inode 55527 errors 100, file extent discount
> Found file extent holes:
>         start: 26738688, len: 462848
> root 161199 inode 55528 errors 100, file extent discount
> Found file extent holes:
>         start: 16252928, len: 184320
> root 161199 inode 55530 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 524288
> root 161199 inode 55531 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 4194304, len: 11001856
>         unresolved ref dir 55511 index 21 name foo filetype 0 errors 3, no dir item, no dir index
> root 161889 inode 54988 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 1572864
> root 161889 inode 54989 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 16252928, len: 91824128
>         unresolved ref dir 54974 index 16 name foo filetype 0 errors 3, no dir item, no dir index
> root 161889 inode 55003 errors 500, file extent discount, nbytes wrong
> Found file extent holes:
>         start: 0, len: 1048576
> root 161889 inode 55004 errors 500, file extent discount, nbytes wrong
> Found file extent holes:
>         start: 0, len: 1048576
>         unresolved ref dir 56235 index 48 name foo filetype mismatch
> root 161889 inode 55409 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 0, len: 524288
>         unresolved ref dir 55399 index 11 name foo filetype 0 errors 3, no dir item, no dir index
> root 161889 inode 55410 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 1572864
> root 161889 inode 55411 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 0, len: 524288
>         unresolved ref dir 55399 index 13 name foo filetype 0 errors 3, no dir item, no dir index
> root 161889 inode 55412 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 0, len: 1048576
>         unresolved ref dir 55399 index 14 name foo filetype 0 errors 3, no dir item, no dir index
> root 161889 inode 55413 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 24117248, len: 34611200
>         unresolved ref dir 55399 index 15 name foo filetype 0 errors 3, no dir item, no dir index
> root 161889 inode 55459 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 56098816, len: 265449472
>         unresolved ref dir 55437 index 23 name foo filetype 0 errors 3, no dir item, no dir index
> root 161889 inode 55468 errors 100, file extent discount
> Found file extent holes:
>         start: 27787264, len: 302174208
> root 161889 inode 55475 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 1048576
> root 161889 inode 55476 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 28803072, len: 459870208
>         unresolved ref dir 55437 index 40 name foo filetype 0 errors 3, no dir item, no dir index
> root 161889 inode 55526 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 524288
> root 161889 inode 55527 errors 100, file extent discount
> Found file extent holes:
>         start: 26738688, len: 462848
> root 161889 inode 55528 errors 100, file extent discount
> Found file extent holes:
>         start: 16252928, len: 184320
> root 161889 inode 55530 errors 100, file extent discount
> Found file extent holes:
>         start: 0, len: 524288
> root 161889 inode 55531 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 4194304, len: 11001856
>         unresolved ref dir 55511 index 21 name foo filetype 0 errors 3, no dir item, no dir index
> root 163920 inode 76551 errors 100, file extent discount
> Found file extent holes:
>         start: 1037262848, len: 184238080
> root 163920 inode 76556 errors 100, file extent discount
> Found file extent holes:
>         start: 1639038976, len: 213225472
>         unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 72785 index 720 name foo filetype mismatch
>         unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 73103 index 544 name foo filetype mismatch
>         unresolved ref dir 73103 index 672 name foo filetype mismatch
>         unresolved ref dir 4179 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 4698 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 5506 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 5546 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
> root 164624 inode 25918 errors 500, file extent discount, nbytes wrong
> Found file extent holes:
>         start: 0, len: 1262850048
> root 164624 inode 72429 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 1333788672, len: 3217154048
>         unresolved ref dir 72438 index 4 name foo filetype 0 errors 3, no dir item, no dir index
> root 164624 inode 72433 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 892338176, len: 14957748224
>         unresolved ref dir 34951 index 13 name foo filetype 0 errors 3, no dir item, no dir index
> root 164624 inode 72593 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 42074112, len: 413642752
>         unresolved ref dir 75036 index 19 name foo filetype 0 errors 3, no dir item, no dir index
>         unresolved ref dir 73103 index 1524 name foo filetype mismatch
> root 164624 inode 73009 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 13869056, len: 53100544
>         unresolved ref dir 72672 index 134 name foo filetype 1 errors 1, no dir item
> root 164624 inode 73083 errors 500, file extent discount, nbytes wrong
> Found file extent holes:
>         start: 197263360, len: 399777792
> root 164624 inode 73094 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 0, len: 524288
>         unresolved ref dir 74963 index 32 name foo filetype 0 errors 3, no dir item, no dir index
> root 164624 inode 73097 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 0, len: 524288
>         unresolved ref dir 74963 index 36 name foo filetype 0 errors 3, no dir item, no dir index
>         unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 73103 index 540 name foo filetype mismatch
>         unresolved ref dir 4179 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 4698 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 5506 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 5546 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
> root 164629 inode 39921 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 759824384, len: 2143326208
>         unresolved ref dir 10205 index 356 name foo filetype 0 errors 3, no dir item, no dir index
> root 164629 inode 72429 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 1333788672, len: 3217154048
>         unresolved ref dir 72438 index 4 name foo filetype 0 errors 3, no dir item, no dir index
> root 164629 inode 72433 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 892338176, len: 14957748224
>         unresolved ref dir 34951 index 13 name foo filetype 0 errors 3, no dir item, no dir index
> root 164629 inode 72593 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 42074112, len: 413642752
>         unresolved ref dir 75036 index 19 name foo filetype 0 errors 3, no dir item, no dir index
>         unresolved ref dir 73103 index 1386 name foo filetype mismatch
> root 164629 inode 73009 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 13869056, len: 53100544
>         unresolved ref dir 72672 index 134 name foo filetype 1 errors 1, no dir item
> root 164629 inode 73083 errors 500, file extent discount, nbytes wrong
> Found file extent holes:
>         start: 197263360, len: 399777792
> root 164629 inode 73094 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 0, len: 524288
>         unresolved ref dir 74963 index 32 name foo filetype 0 errors 3, no dir item, no dir index
> root 164629 inode 73097 errors 2500, file extent discount, nbytes wrong, link count wrong
> Found file extent holes:
>         start: 0, len: 524288
>         unresolved ref dir 74963 index 36 name foo filetype 0 errors 3, no dir item, no dir index
>         unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
>         unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref

Ok the rest of these are going to take some work to fix up.  I'll work
on that as well.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-15 21:18                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
@ 2022-06-15 21:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
  2022-06-15 23:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-15 21:53 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 15, 2022 at 05:18:54PM -0400, Josef Bacik wrote:
> On Wed, Jun 15, 2022 at 10:55 AM Marc MERLIN <marc@merlins.org> wrote:
> >
> > On Wed, Jun 15, 2022 at 07:29:29AM -0700, Marc MERLIN wrote:
> > > gargamel:/mnt/mnt# btrfs scrub start -B .
> > > running that now, I expect it will take a while.
> >
> > Never mind, it was fast:
> > gargamel:/mnt/mnt# btrfs scrub start -B .
> > scrub done for 96539b8c-ccc9-47bf-9e6c-29305890941e
> > Scrub started:    Wed Jun 15 07:28:02 2022
> > Status:           finished
> > Duration:         0:03:33
> > Total to scrub:   111.00GiB
> 
> Hrm shit, this isn't good, don't you have a lot more data than 111gib?
 
Yep, it was closer to 14TB. Ok, so it's probably gone after the many 
commands we ran in the last 2 months.

> Oh oops, I must have missed this in the init-extent-tree.  Let me look
> into this and I'll let you know when you can run the code again.
 
Is there even a reasonable chance to get the data back at this point, or
are we spending effort in not as useful ways?

> Ok the rest of these are going to take some work to fix up.  I'll work
> on that as well.  Thanks,

Up to you, happy to continue if it helps your efforts, but it looks like
my data is mostly gone.
I do remember one command along this thread that had over 100,000 lines
of inodes that were cleared once that bit got automated.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-15 21:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
@ 2022-06-15 23:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
  2022-06-15 23:21                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  0 siblings, 1 reply; 479+ messages in thread
From: Josef Bacik @ 2022-06-15 23:16 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: linux-btrfs

On Wed, Jun 15, 2022 at 5:53 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Wed, Jun 15, 2022 at 05:18:54PM -0400, Josef Bacik wrote:
> > On Wed, Jun 15, 2022 at 10:55 AM Marc MERLIN <marc@merlins.org> wrote:
> > >
> > > On Wed, Jun 15, 2022 at 07:29:29AM -0700, Marc MERLIN wrote:
> > > > gargamel:/mnt/mnt# btrfs scrub start -B .
> > > > running that now, I expect it will take a while.
> > >
> > > Never mind, it was fast:
> > > gargamel:/mnt/mnt# btrfs scrub start -B .
> > > scrub done for 96539b8c-ccc9-47bf-9e6c-29305890941e
> > > Scrub started:    Wed Jun 15 07:28:02 2022
> > > Status:           finished
> > > Duration:         0:03:33
> > > Total to scrub:   111.00GiB
> >
> > Hrm shit, this isn't good, don't you have a lot more data than 111gib?
>
> Yep, it was closer to 14TB. Ok, so it's probably gone after the many
> commands we ran in the last 2 months.
>
> > Oh oops, I must have missed this in the init-extent-tree.  Let me look
> > into this and I'll let you know when you can run the code again.
>
> Is there even a reasonable chance to get the data back at this point, or
> are we spending effort in not as useful ways?
>
> > Ok the rest of these are going to take some work to fix up.  I'll work
> > on that as well.  Thanks,
>
> Up to you, happy to continue if it helps your efforts, but it looks like
> my data is mostly gone.
> I do remember one command along this thread that had over 100,000 lines
> of inodes that were cleared once that bit got automated.
>

Yeah I'm going to go rip that code out.  I should have paid more
attention to what was happening instead of just assuming we had a few
corrupt extents that needed to be removed.

I think we've gotten plenty out of this exercise, sorry I ended up
nuking all of your data.  I know what I need to change to fix these
tools to be more useful later on, and I'll just make a bunch of test
images to validate the work.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-15 23:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
@ 2022-06-15 23:21                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
  2022-06-15 23:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Eldon
  0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2022-06-15 23:21 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs

On Wed, Jun 15, 2022 at 07:16:52PM -0400, Josef Bacik wrote:
> > Up to you, happy to continue if it helps your efforts, but it looks like
> > my data is mostly gone.
> > I do remember one command along this thread that had over 100,000 lines
> > of inodes that were cleared once that bit got automated.
> >
> 
> Yeah I'm going to go rip that code out.  I should have paid more
> attention to what was happening instead of just assuming we had a few
> corrupt extents that needed to be removed.
 
No worries. I knew this was working on live data and that it was
untested code that couuld damage it further :)

> I think we've gotten plenty out of this exercise, sorry I ended up
> nuking all of your data.  I know what I need to change to fix these
> tools to be more useful later on, and I'll just make a bunch of test
> images to validate the work.  Thanks,

Cool, that was the main goal.

Sorry to everyone else following along, hopefully it was somewhat
entertaining :)

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-15 23:21                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
@ 2022-06-15 23:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Eldon
  2022-06-16  0:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Sweet Tea Dorminy
  0 siblings, 1 reply; 479+ messages in thread
From: Eldon @ 2022-06-15 23:26 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: Josef Bacik, linux-btrfs

On Wed, Jun 15, 2022 at 04:21:41PM -0700, Marc MERLIN wrote:
> Sorry to everyone else following along, hopefully it was somewhat
> entertaining :)

You have no idea. Most entertaining thing this year. Popcorn, suspense,
joy, tragedy, redemption.

You really need to put out a tip jar or something.  ;D

Eldon

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-15 23:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Eldon
@ 2022-06-16  0:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Sweet Tea Dorminy
  2022-06-16 20:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Neal Gompa
  0 siblings, 1 reply; 479+ messages in thread
From: Sweet Tea Dorminy @ 2022-06-16  0:22 UTC (permalink / raw)
  To: Eldon, Marc MERLIN; +Cc: Josef Bacik, linux-btrfs

I too found the back-and-forth very educational and suspenseful; many 
thanks for keeping it on the mailing list.

On 6/15/22 19:26, Eldon wrote:
> On Wed, Jun 15, 2022 at 04:21:41PM -0700, Marc MERLIN wrote:
>> Sorry to everyone else following along, hopefully it was somewhat
>> entertaining :)
> 
> You have no idea. Most entertaining thing this year. Popcorn, suspense,
> joy, tragedy, redemption.
> 
> You really need to put out a tip jar or something.  ;D
> 
> Eldon

^ permalink raw reply	[flat|nested] 479+ messages in thread

* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
  2022-06-16  0:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Sweet Tea Dorminy
@ 2022-06-16 20:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Neal Gompa
  0 siblings, 0 replies; 479+ messages in thread
From: Neal Gompa @ 2022-06-16 20:16 UTC (permalink / raw)
  To: Sweet Tea Dorminy; +Cc: Eldon, Marc MERLIN, Josef Bacik, linux-btrfs

On Wed, Jun 15, 2022 at 8:24 PM Sweet Tea Dorminy
<sweettea-kernel@dorminy.me> wrote:
>
> I too found the back-and-forth very educational and suspenseful; many
> thanks for keeping it on the mailing list.
>

It reminded me of my own adventure with Josef a while back. :)


-- 
真実はいつも一つ!/ Always, there's only one truth!

^ permalink raw reply	[flat|nested] 479+ messages in thread

end of thread, other threads:[~2022-06-16 20:17 UTC | newest]

Thread overview: 479+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-07-17 20:32 btrfs check (not lowmem) and OOM-like hangs (4.17.6) Marc MERLIN
2018-07-17 20:59 ` Marc MERLIN
2018-07-18  0:05   ` Qu Wenruo
2018-07-18  0:24     ` Marc MERLIN
2018-07-19  9:22       ` Qu Wenruo
2019-10-18  2:56       ` 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 Marc MERLIN
2019-10-19  3:07         ` Marc MERLIN
2019-10-23  0:55           ` Marc MERLIN
2019-10-26  3:36             ` Marc MERLIN
2020-05-24 21:30       ` 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' Marc MERLIN
2020-05-25  3:01         ` Marc MERLIN
2020-05-25 16:37         ` Chris Murphy
2020-05-25 20:16           ` Marc MERLIN
2020-05-25 20:24             ` Chris Murphy
2020-05-25 20:39               ` Marc MERLIN
2020-05-25 22:47                 ` Chris Murphy
2020-05-25 22:51                   ` Chris Murphy
2020-05-26  0:13                   ` Marc MERLIN
2020-07-07  3:55                 ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN
2020-07-07 14:31                   ` Josef Bacik
2020-07-07 17:25                     ` Marc MERLIN
2020-07-08  3:44                   ` Zygo Blaxell
2020-07-08  4:10                     ` Marc MERLIN
2020-07-08  5:49                       ` Zygo Blaxell
2020-07-08 17:41                         ` Marc MERLIN
2020-07-08 17:44                           ` Roman Mamedov
2020-07-08 22:47                           ` Zygo Blaxell
2022-03-29 17:18                       ` Marc MERLIN
2022-03-30  5:38                         ` Andrei Borzenkov
2022-03-30 14:39                           ` Marc MERLIN
2022-03-31 17:19                             ` Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) Marc MERLIN
2022-04-03 23:33                               ` Josef Bacik
2022-04-04  1:01                                 ` Marc MERLIN
2022-04-04 15:08                                   ` Marc MERLIN
2022-04-04 17:18                                     ` Josef Bacik
2022-04-04 17:43                                       ` Marc MERLIN
2022-04-04 17:53                                         ` Josef Bacik
2022-04-04 18:10                                           ` Marc MERLIN
2022-04-04 18:46                                             ` Josef Bacik
2022-04-04 19:04                                               ` Marc MERLIN
2022-04-04 19:52                                                 ` Josef Bacik
2022-04-04 20:33                                                   ` Marc MERLIN
2022-04-04 21:04                                                     ` Josef Bacik
2022-04-04 21:29                                                       ` Marc MERLIN
2022-04-04 21:40                                                         ` Josef Bacik
2022-04-04 22:09                                                           ` Marc MERLIN
2022-04-04 22:34                                                             ` Josef Bacik
2022-04-04 22:45                                                               ` Marc MERLIN
2022-04-04 22:52                                                                 ` Josef Bacik
2022-04-04 23:18                                                                   ` Marc MERLIN
2022-04-04 23:24                                                                     ` Josef Bacik
2022-04-04 23:42                                                                       ` Marc MERLIN
2022-04-05  0:08                                                                         ` Josef Bacik
2022-04-05  0:13                                                                           ` Marc MERLIN
2022-04-05  0:15                                                                             ` Josef Bacik
2022-04-05  0:18                                                                               ` Marc MERLIN
2022-04-05  0:24                                                                                 ` Josef Bacik
2022-04-05  0:28                                                                                   ` Marc MERLIN
2022-04-05  0:39                                                                                     ` Josef Bacik
2022-04-05  0:58                                                                                       ` Marc MERLIN
2022-04-05  1:06                                                                                         ` Josef Bacik
2022-04-05  1:16                                                                                           ` Marc MERLIN
2022-04-05  1:22                                                                                             ` Josef Bacik
2022-04-05  1:42                                                                                               ` Marc MERLIN
2022-04-05  1:55                                                                                                 ` Josef Bacik
2022-04-05  2:07                                                                                                   ` Marc MERLIN
2022-04-05 14:11                                                                                                     ` Josef Bacik
2022-04-05 15:53                                                                                                       ` Marc MERLIN
2022-04-05 15:55                                                                                                         ` Josef Bacik
2022-04-05 17:41                                                                                                           ` Josef Bacik
2022-04-05 18:11                                                                                                             ` Marc MERLIN
2022-04-05 18:36                                                                                                               ` Josef Bacik
2022-04-05 19:51                                                                                                                 ` Marc MERLIN
2022-04-05 19:56                                                                                                                   ` Josef Bacik
2022-04-05 19:59                                                                                                                     ` Marc MERLIN
2022-04-05 20:05                                                                                                                       ` Josef Bacik
2022-04-05 20:08                                                                                                                         ` Marc MERLIN
2022-04-05 20:24                                                                                                                           ` Josef Bacik
2022-04-05 20:37                                                                                                                             ` Marc MERLIN
2022-04-05 21:07                                                                                                                               ` Josef Bacik
2022-04-05 21:14                                                                                                                                 ` Marc MERLIN
2022-04-05 21:19                                                                                                                                   ` Josef Bacik
2022-04-05 21:25                                                                                                                                     ` Marc MERLIN
2022-04-05 21:26                                                                                                                                     ` Marc MERLIN
2022-04-05 21:35                                                                                                                                       ` Josef Bacik
2022-04-05 21:43                                                                                                                                         ` Marc MERLIN
2022-04-05 22:41                                                                                                                                           ` Josef Bacik
2022-04-05 22:58                                                                                                                                             ` Marc MERLIN
2022-04-06  0:23                                                                                                                                               ` Josef Bacik
2022-04-06  0:30                                                                                                                                                 ` Marc MERLIN
2022-04-06  0:35                                                                                                                                                 ` Marc MERLIN
2022-04-06  0:39                                                                                                                                                   ` Josef Bacik
2022-04-06  1:08                                                                                                                                                     ` Josef Bacik
2022-04-06  1:14                                                                                                                                                       ` Marc MERLIN
2022-04-06  3:12                                                                                                                                                       ` Marc MERLIN
2022-04-06  3:34                                                                                                                                                         ` Marc MERLIN
2022-04-06 15:20                                                                                                                                                           ` Josef Bacik
2022-04-06 18:54                                                                                                                                                             ` Marc MERLIN
2022-04-06 18:57                                                                                                                                                               ` Josef Bacik
2022-04-06 19:16                                                                                                                                                                 ` Marc MERLIN
2022-04-06 19:53                                                                                                                                                                   ` Josef Bacik
2022-04-06 20:34                                                                                                                                                                     ` Marc MERLIN
2022-04-06 20:38                                                                                                                                                                       ` Josef Bacik
2022-04-06 20:56                                                                                                                                                                         ` Marc MERLIN
2022-04-06 21:05                                                                                                                                                                           ` Josef Bacik
2022-04-07  1:08                                                                                                                                                                             ` Marc MERLIN
2022-04-07  1:18                                                                                                                                                                               ` Josef Bacik
2022-04-07  4:37                                                                                                                                                                                 ` Marc MERLIN
2022-04-07  4:40                                                                                                                                                                                   ` Marc MERLIN
2022-04-07  7:30                                                                                                                                                                                     ` Martin Steigerwald
2022-04-07  5:20                                                                                                                                                                                   ` Marc MERLIN
2022-04-07 16:29                                                                                                                                                                                     ` Marc MERLIN
2022-04-07 17:07                                                                                                                                                                                       ` Josef Bacik
2022-04-07 19:11                                                                                                                                                                                         ` Martin Steigerwald
2022-04-07 22:09                                                                                                                                                                                         ` Josef Bacik
2022-04-07 22:33                                                                                                                                                                                           ` Marc MERLIN
2022-04-08 10:22                                                                                                                                                                                           ` Marc MERLIN
2022-04-08 10:23                                                                                                                                                                                             ` Josef Bacik
2022-04-08 20:09                                                                                                                                                                                               ` Josef Bacik
2022-04-11  1:37                                                                                                                                                                                                 ` Marc MERLIN
2022-04-05 23:51                                                                                                                             ` Zygo Blaxell
2022-04-06  0:08                                                                                                                               ` Marc MERLIN
2022-04-06  1:40                                                                                                                                 ` Zygo Blaxell
2022-04-06  4:09                                                                                                                                   ` Marc MERLIN
2022-04-06 18:07                                                                                                                                     ` Zygo Blaxell
2022-04-06 19:13                                                                                                                                       ` Marc MERLIN
2022-04-06 19:45                                                                                                                                         ` Zygo Blaxell
2022-04-06 20:38                                                                                                                                         ` figuring out why transient double raid failure caused a fair amount of btrfs corruption Marc MERLIN
2022-04-06 20:51                                                                                                                                           ` Josef Bacik
2022-04-06 21:14                                                                                                                                             ` Marc MERLIN
2022-04-07 12:27                                                                                                                                             ` Zygo Blaxell
2022-04-22 18:48                                                                                                                                         ` Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) Marc MERLIN
2022-04-22 19:46                                                                                                                                           ` Josef Bacik
2022-04-22 20:01                                                                                                                                             ` Marc MERLIN
2022-04-23 20:12                                                                                                                                               ` Marc MERLIN
2022-04-23 20:53                                                                                                                                                 ` Josef Bacik
2022-04-24 16:20                                                                                                                                                   ` Josef Bacik
2022-04-24 16:24                                                                                                                                                     ` Marc MERLIN
2022-04-24 17:09                                                                                                                                                       ` Josef Bacik
2022-04-24 18:43                                                                                                                                                         ` Marc MERLIN
2022-04-24 19:17                                                                                                                                                           ` Josef Bacik
2022-04-24 19:44                                                                                                                                                             ` Marc MERLIN
2022-04-24 20:01                                                                                                                                                               ` Josef Bacik
2022-04-24 20:31                                                                                                                                                                 ` Marc MERLIN
2022-04-24 20:32                                                                                                                                                                   ` Josef Bacik
2022-04-24 20:54                                                                                                                                                                     ` Marc MERLIN
2022-04-24 21:01                                                                                                                                                                       ` Josef Bacik
     [not found]                                                                                                                                                                         ` <20220424210732.GC29107@merlins.org>
     [not found]                                                                                                                                                                           ` <CAEzrpqcMV+paWShgAnF8d9WaSQ1Fd5R_DZPRQp-+VNsJGDoASg@mail.gmail.com>
     [not found]                                                                                                                                                                             ` <20220424212058.GD29107@merlins.org>
     [not found]                                                                                                                                                                               ` <CAEzrpqcBvh0MC6WeXQ+-80igZhg6t68OcgZnKi6xu+r=njifeA@mail.gmail.com>
2022-04-24 22:38                                                                                                                                                                                 ` Marc MERLIN
2022-04-24 22:56                                                                                                                                                                                   ` Josef Bacik
2022-04-24 23:14                                                                                                                                                                                     ` Marc MERLIN
2022-04-24 23:27                                                                                                                                                                                       ` Josef Bacik
2022-04-25  0:24                                                                                                                                                                                         ` Marc MERLIN
2022-04-25  0:36                                                                                                                                                                                           ` Josef Bacik
2022-04-26  0:28                                                                                                                                                                                             ` Marc MERLIN
2022-04-26 20:43                                                                                                                                                                                               ` Marc MERLIN
2022-04-26 21:20                                                                                                                                                                                                 ` Josef Bacik
2022-04-26 21:36                                                                                                                                                                                                   ` Josef Bacik
2022-04-27  3:54                                                                                                                                                                                                     ` Marc MERLIN
2022-04-27 14:44                                                                                                                                                                                                       ` Josef Bacik
2022-04-27 16:34                                                                                                                                                                                                         ` Marc MERLIN
2022-04-27 17:49                                                                                                                                                                                                           ` Josef Bacik
2022-04-27 18:24                                                                                                                                                                                                             ` Marc MERLIN
2022-04-27 20:21                                                                                                                                                                                                               ` Josef Bacik
2022-04-27 21:02                                                                                                                                                                                                                 ` Marc MERLIN
2022-04-27 21:11                                                                                                                                                                                                                   ` Josef Bacik
2022-04-27 21:20                                                                                                                                                                                                                     ` Marc MERLIN
2022-04-27 21:27                                                                                                                                                                                                                       ` Josef Bacik
2022-04-27 22:59                                                                                                                                                                                                                         ` Marc MERLIN
2022-04-27 23:02                                                                                                                                                                                                                           ` Josef Bacik
2022-04-27 23:21                                                                                                                                                                                                                             ` Josef Bacik
2022-04-28  0:18                                                                                                                                                                                                                               ` Marc MERLIN
2022-04-28  0:44                                                                                                                                                                                                                                 ` Josef Bacik
2022-04-28  3:00                                                                                                                                                                                                                                   ` Marc MERLIN
2022-04-28  3:08                                                                                                                                                                                                                                     ` Josef Bacik
2022-04-28  3:11                                                                                                                                                                                                                                       ` Marc MERLIN
2022-04-28  4:03                                                                                                                                                                                                                                         ` Josef Bacik
2022-04-28  4:12                                                                                                                                                                                                                                           ` Marc MERLIN
2022-04-28 15:30                                                                                                                                                                                                                                             ` Josef Bacik
2022-04-28 16:27                                                                                                                                                                                                                                               ` Marc MERLIN
2022-04-28 20:13                                                                                                                                                                                                                                                 ` Josef Bacik
2022-04-28 20:22                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-04-28 20:28                                                                                                                                                                                                                                                     ` Josef Bacik
2022-04-28 20:57                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-04-28 20:58                                                                                                                                                                                                                                                         ` Josef Bacik
2022-04-28 21:42                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-04-28 21:54                                                                                                                                                                                                                                                             ` Josef Bacik
2022-04-28 22:27                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-04-28 23:24                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-04-29  0:56                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-04-29  1:11                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-04-29  1:34                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-04-29  1:38                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-04-29  4:03                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-04-29 12:41                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-04-29 15:16                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-04-29 15:27                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-04-29 17:16                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-04-29 17:52                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-04-29 18:58                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-04-29 19:40                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-04-30  2:24                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-04-30  3:13                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-04-30 13:07                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-04-30 16:40                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-04-30 23:11                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-05-01  2:48                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-05-01  4:54                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-05-01 11:28                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-05-01 15:22                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-05-01 23:09                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-05-02  1:25                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-05-02 16:41                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-05-02 17:34                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-05-02 19:07                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-05-02 20:08                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-05-02 21:03                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-05-02 21:49                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-05-02 23:16                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-05-02 23:41                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-05-03  1:06                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-05-03  1:26                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-05-03  2:38                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-05-03  4:02                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-05-03  4:13                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-05-03  4:55                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-05-03 16:00                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-05-03 17:24                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-05-05 15:08                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
2022-05-05 15:27                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
2022-05-06  3:19                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
2022-05-07  0:25                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
2022-05-07  1:15                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-05-07 15:39                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-05-07 18:58                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-05-07 19:36                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-05-08 19:45                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
2022-05-08 19:55                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
2022-05-08 20:52                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
2022-05-08 21:20                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-05-08 21:49                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-05-08 22:14                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-05-09  0:22                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-05-09  0:46                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-05-09 16:17                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-05-09 17:00                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-05-09 17:09                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-05-09 17:19                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-05-10  1:04                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-05-10  1:08                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-05-10  1:18                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-05-10  1:32                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-05-10  2:03                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-05-10  2:19                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-05-10 13:21                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-05-10 14:37                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-05-10 15:20                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-05-10 16:06                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-05-10 16:14                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-05-10 16:44                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-05-10 21:15                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
2022-05-10 23:38                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
2022-05-11  0:08                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
2022-05-11  0:28                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
2022-05-11  1:48                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
2022-05-11 11:43                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
2022-05-11 15:03                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
2022-05-11 15:21                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
2022-05-11 16:00                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
2022-05-11 16:05                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
2022-05-11 18:00                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Goffredo Baroncelli
2022-05-12  2:39                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Zygo Blaxell
2022-05-11 18:58                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
     [not found]                                                                                                                                                                                                                                                                                                                                                                                                                                     ` <20220513144113.GA16501@merlins.org>
     [not found]                                                                                                                                                                                                                                                                                                                                                                                                                                       ` <CAEzrpqfYg=Zf_GYjyvc+WZsnoCjiPTAS-08C_rB=gey74DGUqA@mail.gmail.com>
2022-05-15  2:57                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
2022-05-15 14:02                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
2022-05-15 14:41                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
2022-05-15 15:24                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
2022-05-15 15:33                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
2022-05-15 15:35                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
2022-05-15 15:41                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
2022-05-15 15:48                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
2022-05-15 21:29                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
2022-05-15 23:01                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-05-16  0:01                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-05-16  0:57                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-05-16 14:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-05-16 15:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-05-16 15:31                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-05-16 15:36                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
2022-05-16 16:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-05-16 16:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-05-17 19:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
2022-05-17 20:27                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
2022-05-17 20:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
2022-05-17 21:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
2022-05-18 18:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
2022-05-18 19:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
2022-05-18 19:17                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
2022-05-19 22:28                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
2022-05-24  1:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-05-24 18:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-05-24 19:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-05-25 14:35                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-05-26 17:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-05-26 17:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-05-26 17:31                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-05-26 17:44                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-05-26 18:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-05-26 18:54                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-05-26 19:15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-05-26 19:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-05-26 21:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-05-26 23:23                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-05-27  1:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
2022-05-27 18:35                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
2022-05-27 23:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
2022-05-28  0:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
2022-05-28 20:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-05-28 22:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-05-29  1:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-05-29  3:51                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-05-29 15:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-05-29 15:33                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-05-29 17:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-05-29 18:05                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-05-29 18:58                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-05-29 19:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-05-29 19:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-05-29 20:04                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-05-29 20:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-05-30  0:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-05-30  1:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-05-30 19:18                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-05-30 20:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-05-31  1:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-05-31 20:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-05-31 22:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-06-01  0:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-06-01  0:25                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-06-01  1:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-06-01  1:29                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-06-01  2:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-06-01  3:15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-06-01 13:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-06-01 16:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-06-01 18:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-06-01 18:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-06-01 18:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-06-01 18:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-06-01 19:01                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-06-01 20:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
2022-06-01 21:40                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
2022-06-01 22:34                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
2022-06-01 22:36                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
2022-06-01 22:54                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
2022-06-01 22:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
2022-06-01 23:04                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
2022-06-01 23:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
2022-06-02  0:04                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
2022-06-02  0:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
2022-06-02  1:23                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
2022-06-02  1:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
2022-06-02  2:03                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
2022-06-02  2:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
2022-06-02 14:07                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
2022-06-02 14:21                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
2022-06-02 14:27                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
2022-06-02 14:36                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
2022-06-02 18:43                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
2022-06-02 19:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
2022-06-02 19:35                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
2022-06-02 19:51                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
2022-06-02 19:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
2022-06-02 19:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
2022-06-02 20:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
2022-06-02 20:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
2022-06-03  2:20                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Josef Bacik
2022-06-03 14:47                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
2022-06-03 16:17                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Josef Bacik
2022-06-03 16:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
2022-06-03 17:07                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-06-03 18:34                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-06-03 18:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-06-04 12:49                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-06-04 13:48                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-06-04 23:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-06-05  0:13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-06-05 19:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-06-05 20:11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-06-05 20:58                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-06-05 21:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-06-05 21:43                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-06-05 21:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-06-05 23:03                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-06-06  0:05                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-06-06  1:11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-06-06  1:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-06-06 20:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-06-06 21:08                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-06-06 21:19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-06-06 21:23                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-06-06 21:39                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-06-06 21:50                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-06-06 22:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-06-06 22:17                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-06-07  2:28                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-06-07  2:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-06-07  2:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-06-07  3:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-06-07 14:51                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-06-07 14:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-06-07 15:00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-06-07 15:18                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-06-07 15:21                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-06-07 15:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-06-07 17:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-06-07 18:27                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-06-07 19:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-06-07 19:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-06-07 20:10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-06-07 20:25                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-06-07 20:44                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-06-07 20:58                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-06-07 21:25                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-06-07 23:33                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-06-07 23:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-06-07 23:41                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-06-08  0:07                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-06-08  0:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-06-08  0:42                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-06-08  1:31                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-06-08  2:12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-06-08 20:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-06-08 21:30                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-06-08 21:33                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-06-08 21:38                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-06-08 22:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-06-09  3:01                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-06-09 20:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-06-09 21:15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-06-10 18:47                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-06-10 19:11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Marc MERLIN
2022-06-10 19:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Josef Bacik
2022-06-11  0:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-06-11 14:59                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-06-12 17:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-06-12 20:05                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-06-12 21:19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-06-12 22:32                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-06-12 17:37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-06-12 20:06                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-06-12 21:14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-06-13 17:56                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Marc MERLIN
2022-06-13 18:28                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Marc MERLIN
2022-06-13 18:29                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Josef Bacik
2022-06-13 20:46                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-06-13 22:19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Josef Bacik
2022-06-13 23:52                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Marc MERLIN
2022-06-15  1:44                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Josef Bacik
2022-06-15 14:29                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Marc MERLIN
2022-06-15 14:55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ` Marc MERLIN
2022-06-15 21:18                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ` Josef Bacik
2022-06-15 21:53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ` Marc MERLIN
2022-06-15 23:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ` Josef Bacik
2022-06-15 23:21                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ` Marc MERLIN
2022-06-15 23:26                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ` Eldon
2022-06-16  0:22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ` Sweet Tea Dorminy
2022-06-16 20:16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ` Neal Gompa
2022-04-05 16:22                                                                                                 ` Roman Mamedov
2022-04-05 22:06                                                                                                   ` Marc MERLIN
2022-04-05 18:38                                                                                                 ` Zygo Blaxell
2022-04-05 19:31                                                                                                   ` Marc MERLIN
2020-08-12 22:34                     ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN
2020-08-13  7:39                       ` Roman Mamedov
2020-08-13 15:07                         ` Marc MERLIN
2020-08-14  2:19                           ` Zygo Blaxell
2020-08-14  1:43                       ` Zygo Blaxell
2020-08-15  4:41                         ` Marc MERLIN
2018-07-18 19:42     ` btrfs check (not lowmem) and OOM-like hangs (4.17.6) Andrei Borzenkov
2018-07-18 21:56       ` Marc MERLIN

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.