* recovering from "parent transid verify failed" @ 2019-08-14 18:32 Tim Walberg 2019-08-15 2:35 ` Qu Wenruo 0 siblings, 1 reply; 8+ messages in thread From: Tim Walberg @ 2019-08-14 18:32 UTC (permalink / raw) To: linux-btrfs Most of the recommendations I've found online deal with when "wanted" is greater than "found", which, if I understand correctly means that one or more transactions were interrupted/lost before fully committed. Are the recommendations for recovery the same if the system is reporting a "wanted" that is less than "found"? -- twalberg@comcast.net ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: recovering from "parent transid verify failed" 2019-08-14 18:32 recovering from "parent transid verify failed" Tim Walberg @ 2019-08-15 2:35 ` Qu Wenruo 2019-08-15 13:52 ` Tim Walberg 2019-08-15 13:55 ` Tim Walberg 0 siblings, 2 replies; 8+ messages in thread From: Qu Wenruo @ 2019-08-15 2:35 UTC (permalink / raw) To: Tim Walberg, linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 842 bytes --] On 2019/8/15 上午2:32, Tim Walberg wrote: > Most of the recommendations I've found online deal with when "wanted" is > greater than "found", which, if I understand correctly means that one or > more transactions were interrupted/lost before fully committed. No matter what the case is, a proper transaction shouldn't have any tree block overwritten. That means, either the FLUSH/FUA of the hardware/lower block layer is screwed up, or the COW of tree block is already screwed up. > > Are the recommendations for recovery the same if the system is reporting a > "wanted" that is less than "found"? > The salvage is no difference than any transid mismatch, no matter if it's larger or smaller. It depends on the tree block. Please provide full dmesg output and btrfs check for further advice. Thanks, Qu [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: recovering from "parent transid verify failed" 2019-08-15 2:35 ` Qu Wenruo @ 2019-08-15 13:52 ` Tim Walberg 2019-08-15 14:07 ` Qu Wenruo 2019-08-15 13:55 ` Tim Walberg 1 sibling, 1 reply; 8+ messages in thread From: Tim Walberg @ 2019-08-15 13:52 UTC (permalink / raw) To: Qu Wenruo; +Cc: Tim Walberg, linux-btrfs Had to wait for 'btrfs recover' to finish before I proceed farther. Kernel is 4.19.45, tools are 4.19.1 File system is a 3-disk RAID10 with WD3003FZEX (WD Black 3TB) Output from attempting to mount: # mount -o ro,usebackuproot /dev/sdc1 /mnt mount: wrong fs type, bad option, bad superblock on /dev/sdc1, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. Kernel messages from the mount attempt: [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): trying to use backup root at mount time [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): disk space caching is enabled [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): has skinny extents [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750 [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750 [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): failed to read block groups: -5 [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): open_ctree failed Output from 'btrfs check -p /dev/sdc1': # btrfs check -p /dev/sdc1 Opening filesystem to check... parent transid verify failed on 229846466560 wanted 49749 found 49750 Ignoring transid failure ERROR: child eb corrupted: parent bytenr=229845336064 item=0 parent level=1 child level=2 ERROR: cannot open file system On 08/15/2019 10:35 +0800, Qu Wenruo wrote: >> >> >> On 2019/8/15 ??????2:32, Tim Walberg wrote: >> > Most of the recommendations I've found online deal with when "wanted" is >> > greater than "found", which, if I understand correctly means that one or >> > more transactions were interrupted/lost before fully committed. >> >> No matter what the case is, a proper transaction shouldn't have any tree >> block overwritten. >> >> That means, either the FLUSH/FUA of the hardware/lower block layer is >> screwed up, or the COW of tree block is already screwed up. >> >> > >> > Are the recommendations for recovery the same if the system is reporting a >> > "wanted" that is less than "found"? >> > >> The salvage is no difference than any transid mismatch, no matter if >> it's larger or smaller. >> >> It depends on the tree block. >> >> Please provide full dmesg output and btrfs check for further advice. >> >> Thanks, >> Qu >> -- +----------------------+ | Tim Walberg | | 830 Carriage Dr. | | Algonquin, IL 60102 | | twalberg@comcast.net | +----------------------+ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: recovering from "parent transid verify failed" 2019-08-15 13:52 ` Tim Walberg @ 2019-08-15 14:07 ` Qu Wenruo 2019-08-15 14:21 ` Tim Walberg 0 siblings, 1 reply; 8+ messages in thread From: Qu Wenruo @ 2019-08-15 14:07 UTC (permalink / raw) To: Tim Walberg; +Cc: linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 3447 bytes --] On 2019/8/15 下午9:52, Tim Walberg wrote: > Had to wait for 'btrfs recover' to finish before I proceed farther. > > Kernel is 4.19.45, tools are 4.19.1 > > File system is a 3-disk RAID10 with WD3003FZEX (WD Black 3TB) > > Output from attempting to mount: > > # mount -o ro,usebackuproot /dev/sdc1 /mnt > mount: wrong fs type, bad option, bad superblock on /dev/sdc1, > missing codepage or helper program, or other error > > In some cases useful info is found in syslog - try > dmesg | tail or so. > > Kernel messages from the mount attempt: > > [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): trying to use backup root at mount time > [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): disk space caching is enabled > [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): has skinny extents > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750 > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750 > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): failed to read block groups: -5 Extent tree corruption. So if that's the only corruption, you have a very high chance to recover most of your data. Btrfs rescue can work, or you can try the experimental patches which provides rescue=skip_bg mount option to allow you mount the fs RO and receive your data (later is way faster than user space rescue) https://patchwork.kernel.org/project/linux-btrfs/list/?series=130637 Also, for your dump super output, it doesn't provide too much info. You would like to use -Ffa option for more info. Also, you could also try that on all 3 devices, to find out which one has lower generation. Also, please provide the history of the corruption. One generation corruptions is a little rare. Is sudden power loss involved in this case? Thanks, Qu > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): open_ctree failed > > Output from 'btrfs check -p /dev/sdc1': > > # btrfs check -p /dev/sdc1 > Opening filesystem to check... > parent transid verify failed on 229846466560 wanted 49749 found 49750 > Ignoring transid failure > ERROR: child eb corrupted: parent bytenr=229845336064 item=0 parent level=1 child level=2 > ERROR: cannot open file system > > > > On 08/15/2019 10:35 +0800, Qu Wenruo wrote: >>> >>> >>> On 2019/8/15 ??????2:32, Tim Walberg wrote: >>> > Most of the recommendations I've found online deal with when "wanted" is >>> > greater than "found", which, if I understand correctly means that one or >>> > more transactions were interrupted/lost before fully committed. >>> >>> No matter what the case is, a proper transaction shouldn't have any tree >>> block overwritten. >>> >>> That means, either the FLUSH/FUA of the hardware/lower block layer is >>> screwed up, or the COW of tree block is already screwed up. >>> >>> > >>> > Are the recommendations for recovery the same if the system is reporting a >>> > "wanted" that is less than "found"? >>> > >>> The salvage is no difference than any transid mismatch, no matter if >>> it's larger or smaller. >>> >>> It depends on the tree block. >>> >>> Please provide full dmesg output and btrfs check for further advice. >>> >>> Thanks, >>> Qu >>> > > > > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: recovering from "parent transid verify failed" 2019-08-15 14:07 ` Qu Wenruo @ 2019-08-15 14:21 ` Tim Walberg 2019-08-15 14:45 ` Qu Wenruo 0 siblings, 1 reply; 8+ messages in thread From: Tim Walberg @ 2019-08-15 14:21 UTC (permalink / raw) To: Qu Wenruo; +Cc: Tim Walberg, linux-btrfs [-- Attachment #1: Type: text/plain, Size: 4413 bytes --] 'dump-super -Ffa' from all three devices attached. 'btrfs restore' did appear to recover most of the main data, minus snapshots, which would have greatly increased the required time and capacity, since I was recovering to XFS. 'btrfs rescue chunk-recover' ran, but failed to fix anything. 'btrfs rescue super-recover' says all supers are fine. Initial corruption was due to a hard hang, which didn't leave enough crumbs to determine the source - might have been btrfs, might have been nvidia, might have been something completely different. On 08/15/2019 22:07 +0800, Qu Wenruo wrote: >> >> >> On 2019/8/15 ??????9:52, Tim Walberg wrote: >> > Had to wait for 'btrfs recover' to finish before I proceed farther. >> > >> > Kernel is 4.19.45, tools are 4.19.1 >> > >> > File system is a 3-disk RAID10 with WD3003FZEX (WD Black 3TB) >> > >> > Output from attempting to mount: >> > >> > # mount -o ro,usebackuproot /dev/sdc1 /mnt >> > mount: wrong fs type, bad option, bad superblock on /dev/sdc1, >> > missing codepage or helper program, or other error >> > >> > In some cases useful info is found in syslog - try >> > dmesg | tail or so. >> > >> > Kernel messages from the mount attempt: >> > >> > [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): trying to use backup root at mount time >> > [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): disk space caching is enabled >> > [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): has skinny extents >> > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750 >> > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750 >> > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): failed to read block groups: -5 >> >> Extent tree corruption. >> >> So if that's the only corruption, you have a very high chance to recover >> most of your data. >> >> Btrfs rescue can work, or you can try the experimental patches which >> provides rescue=skip_bg mount option to allow you mount the fs RO and >> receive your data (later is way faster than user space rescue) >> https://patchwork.kernel.org/project/linux-btrfs/list/?series=130637 >> >> Also, for your dump super output, it doesn't provide too much info. >> >> You would like to use -Ffa option for more info. >> Also, you could also try that on all 3 devices, to find out which one >> has lower generation. >> >> Also, please provide the history of the corruption. >> One generation corruptions is a little rare. Is sudden power loss >> involved in this case? >> >> Thanks, >> Qu >> >> > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): open_ctree failed >> > >> > Output from 'btrfs check -p /dev/sdc1': >> > >> > # btrfs check -p /dev/sdc1 >> > Opening filesystem to check... >> > parent transid verify failed on 229846466560 wanted 49749 found 49750 >> > Ignoring transid failure >> > ERROR: child eb corrupted: parent bytenr=229845336064 item=0 parent level=1 child level=2 >> > ERROR: cannot open file system >> > >> > >> > >> > On 08/15/2019 10:35 +0800, Qu Wenruo wrote: >> >>> >> >>> >> >>> On 2019/8/15 ??????2:32, Tim Walberg wrote: >> >>> > Most of the recommendations I've found online deal with when "wanted" is >> >>> > greater than "found", which, if I understand correctly means that one or >> >>> > more transactions were interrupted/lost before fully committed. >> >>> >> >>> No matter what the case is, a proper transaction shouldn't have any tree >> >>> block overwritten. >> >>> >> >>> That means, either the FLUSH/FUA of the hardware/lower block layer is >> >>> screwed up, or the COW of tree block is already screwed up. >> >>> >> >>> > >> >>> > Are the recommendations for recovery the same if the system is reporting a >> >>> > "wanted" that is less than "found"? >> >>> > >> >>> The salvage is no difference than any transid mismatch, no matter if >> >>> it's larger or smaller. >> >>> >> >>> It depends on the tree block. >> >>> >> >>> Please provide full dmesg output and btrfs check for further advice. >> >>> >> >>> Thanks, >> >>> Qu >> >>> >> > >> > >> > >> > >> End of included message -- +----------------------+ | Tim Walberg | | 830 Carriage Dr. | | Algonquin, IL 60102 | | twalberg@comcast.net | +----------------------+ [-- Attachment #2: supers.txt --] [-- Type: text/plain, Size: 30165 bytes --] superblock: bytenr=65536, device=/dev/sdc1 --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0x4331039b [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 53749823-faaf-46f9-866d-3778d93cb1ca label btrfs1 generation 49750 root 229846646784 sys_array_size 129 chunk_root_generation 49725 root_level 1 chunk_root 2568857059328 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 9001775738880 bytes_used 2085801975808 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 3 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x361 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA | NO_HOLES ) cache_generation 49748 uuid_tree_generation 49748 dev_item.uuid 7338a973-9a45-4032-a4c9-d18142fd7908 dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match] dev_item.type 0 dev_item.total_bytes 3000591912960 dev_item.bytes_used 1407675531264 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328) length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1 io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 2 offset 300690702336 dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562 stripe 1 devid 3 offset 1083179008 dev_uuid ee71c327-ff63-4f46-8177-6328976f891f backup_roots[4]: backup 0: backup_tree_root: 205750272 gen: 49747 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 174325760 gen: 49747 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 270893056 gen: 49747 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 1: backup_tree_root: 230054100992 gen: 49748 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 230054117376 gen: 49748 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 230736379904 gen: 49748 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 2: backup_tree_root: 2031969501184 gen: 49745 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031960915968 gen: 49745 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2031960244224 gen: 49745 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802795008 backup_num_devices: 3 backup 3: backup_tree_root: 2032017784832 gen: 49746 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031988506624 gen: 49746 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2032023994368 gen: 49746 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802778624 backup_num_devices: 3 superblock: bytenr=67108864, device=/dev/sdc1 --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0xe3502b55 [match] bytenr 67108864 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 53749823-faaf-46f9-866d-3778d93cb1ca label btrfs1 generation 49750 root 229846646784 sys_array_size 129 chunk_root_generation 49725 root_level 1 chunk_root 2568857059328 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 9001775738880 bytes_used 2085801975808 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 3 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x361 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA | NO_HOLES ) cache_generation 49748 uuid_tree_generation 49748 dev_item.uuid 7338a973-9a45-4032-a4c9-d18142fd7908 dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match] dev_item.type 0 dev_item.total_bytes 3000591912960 dev_item.bytes_used 1407675531264 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328) length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1 io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 2 offset 300690702336 dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562 stripe 1 devid 3 offset 1083179008 dev_uuid ee71c327-ff63-4f46-8177-6328976f891f backup_roots[4]: backup 0: backup_tree_root: 205750272 gen: 49747 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 174325760 gen: 49747 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 270893056 gen: 49747 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 1: backup_tree_root: 230054100992 gen: 49748 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 230054117376 gen: 49748 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 230736379904 gen: 49748 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 2: backup_tree_root: 2031969501184 gen: 49745 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031960915968 gen: 49745 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2031960244224 gen: 49745 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802795008 backup_num_devices: 3 backup 3: backup_tree_root: 2032017784832 gen: 49746 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031988506624 gen: 49746 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2032023994368 gen: 49746 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802778624 backup_num_devices: 3 superblock: bytenr=274877906944, device=/dev/sdc1 --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0x1ed77d64 [match] bytenr 274877906944 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 53749823-faaf-46f9-866d-3778d93cb1ca label btrfs1 generation 49750 root 229846646784 sys_array_size 129 chunk_root_generation 49725 root_level 1 chunk_root 2568857059328 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 9001775738880 bytes_used 2085801975808 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 3 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x361 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA | NO_HOLES ) cache_generation 49748 uuid_tree_generation 49748 dev_item.uuid 7338a973-9a45-4032-a4c9-d18142fd7908 dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match] dev_item.type 0 dev_item.total_bytes 3000591912960 dev_item.bytes_used 1407675531264 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328) length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1 io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 2 offset 300690702336 dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562 stripe 1 devid 3 offset 1083179008 dev_uuid ee71c327-ff63-4f46-8177-6328976f891f backup_roots[4]: backup 0: backup_tree_root: 205750272 gen: 49747 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 174325760 gen: 49747 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 270893056 gen: 49747 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 1: backup_tree_root: 230054100992 gen: 49748 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 230054117376 gen: 49748 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 230736379904 gen: 49748 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 2: backup_tree_root: 2031969501184 gen: 49745 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031960915968 gen: 49745 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2031960244224 gen: 49745 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802795008 backup_num_devices: 3 backup 3: backup_tree_root: 2032017784832 gen: 49746 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031988506624 gen: 49746 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2032023994368 gen: 49746 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802778624 backup_num_devices: 3 superblock: bytenr=65536, device=/dev/sdd1 --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0xd0f74999 [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 53749823-faaf-46f9-866d-3778d93cb1ca label btrfs1 generation 49750 root 229846646784 sys_array_size 129 chunk_root_generation 49725 root_level 1 chunk_root 2568857059328 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 9001775738880 bytes_used 2085801975808 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 3 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x361 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA | NO_HOLES ) cache_generation 49748 uuid_tree_generation 49748 dev_item.uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562 dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match] dev_item.type 0 dev_item.total_bytes 3000591912960 dev_item.bytes_used 1404487860224 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 2 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328) length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1 io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 2 offset 300690702336 dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562 stripe 1 devid 3 offset 1083179008 dev_uuid ee71c327-ff63-4f46-8177-6328976f891f backup_roots[4]: backup 0: backup_tree_root: 205750272 gen: 49747 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 174325760 gen: 49747 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 270893056 gen: 49747 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 1: backup_tree_root: 230054100992 gen: 49748 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 230054117376 gen: 49748 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 230736379904 gen: 49748 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 2: backup_tree_root: 2031969501184 gen: 49745 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031960915968 gen: 49745 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2031960244224 gen: 49745 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802795008 backup_num_devices: 3 backup 3: backup_tree_root: 2032017784832 gen: 49746 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031988506624 gen: 49746 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2032023994368 gen: 49746 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802778624 backup_num_devices: 3 superblock: bytenr=67108864, device=/dev/sdd1 --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0x70966157 [match] bytenr 67108864 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 53749823-faaf-46f9-866d-3778d93cb1ca label btrfs1 generation 49750 root 229846646784 sys_array_size 129 chunk_root_generation 49725 root_level 1 chunk_root 2568857059328 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 9001775738880 bytes_used 2085801975808 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 3 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x361 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA | NO_HOLES ) cache_generation 49748 uuid_tree_generation 49748 dev_item.uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562 dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match] dev_item.type 0 dev_item.total_bytes 3000591912960 dev_item.bytes_used 1404487860224 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 2 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328) length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1 io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 2 offset 300690702336 dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562 stripe 1 devid 3 offset 1083179008 dev_uuid ee71c327-ff63-4f46-8177-6328976f891f backup_roots[4]: backup 0: backup_tree_root: 205750272 gen: 49747 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 174325760 gen: 49747 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 270893056 gen: 49747 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 1: backup_tree_root: 230054100992 gen: 49748 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 230054117376 gen: 49748 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 230736379904 gen: 49748 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 2: backup_tree_root: 2031969501184 gen: 49745 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031960915968 gen: 49745 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2031960244224 gen: 49745 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802795008 backup_num_devices: 3 backup 3: backup_tree_root: 2032017784832 gen: 49746 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031988506624 gen: 49746 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2032023994368 gen: 49746 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802778624 backup_num_devices: 3 superblock: bytenr=274877906944, device=/dev/sdd1 --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0x8d113766 [match] bytenr 274877906944 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 53749823-faaf-46f9-866d-3778d93cb1ca label btrfs1 generation 49750 root 229846646784 sys_array_size 129 chunk_root_generation 49725 root_level 1 chunk_root 2568857059328 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 9001775738880 bytes_used 2085801975808 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 3 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x361 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA | NO_HOLES ) cache_generation 49748 uuid_tree_generation 49748 dev_item.uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562 dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match] dev_item.type 0 dev_item.total_bytes 3000591912960 dev_item.bytes_used 1404487860224 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 2 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328) length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1 io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 2 offset 300690702336 dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562 stripe 1 devid 3 offset 1083179008 dev_uuid ee71c327-ff63-4f46-8177-6328976f891f backup_roots[4]: backup 0: backup_tree_root: 205750272 gen: 49747 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 174325760 gen: 49747 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 270893056 gen: 49747 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 1: backup_tree_root: 230054100992 gen: 49748 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 230054117376 gen: 49748 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 230736379904 gen: 49748 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 2: backup_tree_root: 2031969501184 gen: 49745 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031960915968 gen: 49745 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2031960244224 gen: 49745 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802795008 backup_num_devices: 3 backup 3: backup_tree_root: 2032017784832 gen: 49746 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031988506624 gen: 49746 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2032023994368 gen: 49746 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802778624 backup_num_devices: 3 superblock: bytenr=65536, device=/dev/sde1 --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0xb89a7a2e [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 53749823-faaf-46f9-866d-3778d93cb1ca label btrfs1 generation 49750 root 229846646784 sys_array_size 129 chunk_root_generation 49725 root_level 1 chunk_root 2568857059328 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 9001775738880 bytes_used 2085801975808 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 3 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x361 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA | NO_HOLES ) cache_generation 49748 uuid_tree_generation 49748 dev_item.uuid ee71c327-ff63-4f46-8177-6328976f891f dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match] dev_item.type 0 dev_item.total_bytes 3000591912960 dev_item.bytes_used 1405561602048 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 3 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328) length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1 io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 2 offset 300690702336 dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562 stripe 1 devid 3 offset 1083179008 dev_uuid ee71c327-ff63-4f46-8177-6328976f891f backup_roots[4]: backup 0: backup_tree_root: 205750272 gen: 49747 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 174325760 gen: 49747 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 270893056 gen: 49747 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 1: backup_tree_root: 230054100992 gen: 49748 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 230054117376 gen: 49748 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 230736379904 gen: 49748 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 2: backup_tree_root: 2031969501184 gen: 49745 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031960915968 gen: 49745 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2031960244224 gen: 49745 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802795008 backup_num_devices: 3 backup 3: backup_tree_root: 2032017784832 gen: 49746 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031988506624 gen: 49746 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2032023994368 gen: 49746 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802778624 backup_num_devices: 3 superblock: bytenr=67108864, device=/dev/sde1 --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0x18fb52e0 [match] bytenr 67108864 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 53749823-faaf-46f9-866d-3778d93cb1ca label btrfs1 generation 49750 root 229846646784 sys_array_size 129 chunk_root_generation 49725 root_level 1 chunk_root 2568857059328 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 9001775738880 bytes_used 2085801975808 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 3 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x361 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA | NO_HOLES ) cache_generation 49748 uuid_tree_generation 49748 dev_item.uuid ee71c327-ff63-4f46-8177-6328976f891f dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match] dev_item.type 0 dev_item.total_bytes 3000591912960 dev_item.bytes_used 1405561602048 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 3 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328) length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1 io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 2 offset 300690702336 dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562 stripe 1 devid 3 offset 1083179008 dev_uuid ee71c327-ff63-4f46-8177-6328976f891f backup_roots[4]: backup 0: backup_tree_root: 205750272 gen: 49747 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 174325760 gen: 49747 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 270893056 gen: 49747 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 1: backup_tree_root: 230054100992 gen: 49748 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 230054117376 gen: 49748 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 230736379904 gen: 49748 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 2: backup_tree_root: 2031969501184 gen: 49745 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031960915968 gen: 49745 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2031960244224 gen: 49745 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802795008 backup_num_devices: 3 backup 3: backup_tree_root: 2032017784832 gen: 49746 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031988506624 gen: 49746 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2032023994368 gen: 49746 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802778624 backup_num_devices: 3 superblock: bytenr=274877906944, device=/dev/sde1 --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0xe57c04d1 [match] bytenr 274877906944 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 53749823-faaf-46f9-866d-3778d93cb1ca label btrfs1 generation 49750 root 229846646784 sys_array_size 129 chunk_root_generation 49725 root_level 1 chunk_root 2568857059328 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 9001775738880 bytes_used 2085801975808 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 3 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x361 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA | NO_HOLES ) cache_generation 49748 uuid_tree_generation 49748 dev_item.uuid ee71c327-ff63-4f46-8177-6328976f891f dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match] dev_item.type 0 dev_item.total_bytes 3000591912960 dev_item.bytes_used 1405561602048 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 3 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328) length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1 io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 2 offset 300690702336 dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562 stripe 1 devid 3 offset 1083179008 dev_uuid ee71c327-ff63-4f46-8177-6328976f891f backup_roots[4]: backup 0: backup_tree_root: 205750272 gen: 49747 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 174325760 gen: 49747 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 270893056 gen: 49747 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 1: backup_tree_root: 230054100992 gen: 49748 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 230054117376 gen: 49748 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 229931761664 gen: 49747 level: 1 backup_csum_root: 230736379904 gen: 49748 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085801975808 backup_num_devices: 3 backup 2: backup_tree_root: 2031969501184 gen: 49745 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031960915968 gen: 49745 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2031960244224 gen: 49745 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802795008 backup_num_devices: 3 backup 3: backup_tree_root: 2032017784832 gen: 49746 level: 1 backup_chunk_root: 2568857059328 gen: 49725 level: 1 backup_extent_root: 2031988506624 gen: 49746 level: 2 backup_fs_root: 850472665088 gen: 49727 level: 0 backup_dev_root: 230821036032 gen: 49738 level: 1 backup_csum_root: 2032023994368 gen: 49746 level: 3 backup_total_bytes: 9001775738880 backup_bytes_used: 2085802778624 backup_num_devices: 3 ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: recovering from "parent transid verify failed" 2019-08-15 14:21 ` Tim Walberg @ 2019-08-15 14:45 ` Qu Wenruo 2019-08-15 15:01 ` Tim Walberg 0 siblings, 1 reply; 8+ messages in thread From: Qu Wenruo @ 2019-08-15 14:45 UTC (permalink / raw) To: Tim Walberg; +Cc: linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 5802 bytes --] On 2019/8/15 下午10:21, Tim Walberg wrote: > 'dump-super -Ffa' from all three devices attached. > > 'btrfs restore' did appear to recover most of the main data, minus > snapshots, which would have greatly increased the required time and > capacity, since I was recovering to XFS. That's why I recommend that experimental patchset, it will make the fs mountable (RO though), with all btrfs snapshots available. > > 'btrfs rescue chunk-recover' ran, but failed to fix anything. > 'btrfs rescue super-recover' says all supers are fine. Those are useless for your case. > > Initial corruption was due to a hard hang, which didn't leave enough > crumbs to determine the source - might have been btrfs, might have > been nvidia, might have been something completely different. Anyway, the corruption is a little strange. First of all, even hard hang/power loss shouldn't cause btrfs to overwrite its tree block, thus even hard hang/power loss happens, btrfs should be corrupted. But that's definitely not the case. (We have quite some such report, but haven't pinned down the cause yet) Secondly, the generation of your fs is strange. The latest geneartion of your tree root is 49750, matches with your corrupted tree block, but your extent tree is definitely older. So it looks like, your super blocks (all nine!) reach disk before some tree blocks reach the disk. Finally, the superblock doesn't record previous transaction correctly. It doesn't has transaction of 49749 in its backup roots. Not 100% sure, but looks somewhat like the problem fixed by this patch: Btrfs: fix race leading to fs corruption after transaction abortion It should get backported to all stable release recently. Thanks, Qu > > > On 08/15/2019 22:07 +0800, Qu Wenruo wrote: >>> >>> >>> On 2019/8/15 ??????9:52, Tim Walberg wrote: >>> > Had to wait for 'btrfs recover' to finish before I proceed farther. >>> > >>> > Kernel is 4.19.45, tools are 4.19.1 >>> > >>> > File system is a 3-disk RAID10 with WD3003FZEX (WD Black 3TB) >>> > >>> > Output from attempting to mount: >>> > >>> > # mount -o ro,usebackuproot /dev/sdc1 /mnt >>> > mount: wrong fs type, bad option, bad superblock on /dev/sdc1, >>> > missing codepage or helper program, or other error >>> > >>> > In some cases useful info is found in syslog - try >>> > dmesg | tail or so. >>> > >>> > Kernel messages from the mount attempt: >>> > >>> > [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): trying to use backup root at mount time >>> > [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): disk space caching is enabled >>> > [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): has skinny extents >>> > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750 >>> > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750 >>> > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): failed to read block groups: -5 >>> >>> Extent tree corruption. >>> >>> So if that's the only corruption, you have a very high chance to recover >>> most of your data. >>> >>> Btrfs rescue can work, or you can try the experimental patches which >>> provides rescue=skip_bg mount option to allow you mount the fs RO and >>> receive your data (later is way faster than user space rescue) >>> https://patchwork.kernel.org/project/linux-btrfs/list/?series=130637 >>> >>> Also, for your dump super output, it doesn't provide too much info. >>> >>> You would like to use -Ffa option for more info. >>> Also, you could also try that on all 3 devices, to find out which one >>> has lower generation. >>> >>> Also, please provide the history of the corruption. >>> One generation corruptions is a little rare. Is sudden power loss >>> involved in this case? >>> >>> Thanks, >>> Qu >>> >>> > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): open_ctree failed >>> > >>> > Output from 'btrfs check -p /dev/sdc1': >>> > >>> > # btrfs check -p /dev/sdc1 >>> > Opening filesystem to check... >>> > parent transid verify failed on 229846466560 wanted 49749 found 49750 >>> > Ignoring transid failure >>> > ERROR: child eb corrupted: parent bytenr=229845336064 item=0 parent level=1 child level=2 >>> > ERROR: cannot open file system >>> > >>> > >>> > >>> > On 08/15/2019 10:35 +0800, Qu Wenruo wrote: >>> >>> >>> >>> >>> >>> On 2019/8/15 ??????2:32, Tim Walberg wrote: >>> >>> > Most of the recommendations I've found online deal with when "wanted" is >>> >>> > greater than "found", which, if I understand correctly means that one or >>> >>> > more transactions were interrupted/lost before fully committed. >>> >>> >>> >>> No matter what the case is, a proper transaction shouldn't have any tree >>> >>> block overwritten. >>> >>> >>> >>> That means, either the FLUSH/FUA of the hardware/lower block layer is >>> >>> screwed up, or the COW of tree block is already screwed up. >>> >>> >>> >>> > >>> >>> > Are the recommendations for recovery the same if the system is reporting a >>> >>> > "wanted" that is less than "found"? >>> >>> > >>> >>> The salvage is no difference than any transid mismatch, no matter if >>> >>> it's larger or smaller. >>> >>> >>> >>> It depends on the tree block. >>> >>> >>> >>> Please provide full dmesg output and btrfs check for further advice. >>> >>> >>> >>> Thanks, >>> >>> Qu >>> >>> >>> > >>> > >>> > >>> > >>> > > > > End of included message > > > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: recovering from "parent transid verify failed" 2019-08-15 14:45 ` Qu Wenruo @ 2019-08-15 15:01 ` Tim Walberg 0 siblings, 0 replies; 8+ messages in thread From: Tim Walberg @ 2019-08-15 15:01 UTC (permalink / raw) To: Qu Wenruo; +Cc: Tim Walberg, linux-btrfs Thanks for all the help! If I get a chance later today, I may try the patch set, but in the interest of getting things back online quicker, I may just have to recreate and restore the recovered data. The snapshots are no great loss - they're just one level of daily backups. tw On 08/15/2019 22:45 +0800, Qu Wenruo wrote: >> >> >> On 2019/8/15 ??????10:21, Tim Walberg wrote: >> > 'dump-super -Ffa' from all three devices attached. >> > >> > 'btrfs restore' did appear to recover most of the main data, minus >> > snapshots, which would have greatly increased the required time and >> > capacity, since I was recovering to XFS. >> >> That's why I recommend that experimental patchset, it will make the fs >> mountable (RO though), with all btrfs snapshots available. >> >> > >> > 'btrfs rescue chunk-recover' ran, but failed to fix anything. >> > 'btrfs rescue super-recover' says all supers are fine. >> >> Those are useless for your case. >> >> > >> > Initial corruption was due to a hard hang, which didn't leave enough >> > crumbs to determine the source - might have been btrfs, might have >> > been nvidia, might have been something completely different. >> >> Anyway, the corruption is a little strange. >> >> First of all, even hard hang/power loss shouldn't cause btrfs to >> overwrite its tree block, thus even hard hang/power loss happens, btrfs >> should be corrupted. >> >> But that's definitely not the case. (We have quite some such report, but >> haven't pinned down the cause yet) >> >> Secondly, the generation of your fs is strange. >> The latest geneartion of your tree root is 49750, matches with your >> corrupted tree block, but your extent tree is definitely older. >> >> So it looks like, your super blocks (all nine!) reach disk before some >> tree blocks reach the disk. >> >> Finally, the superblock doesn't record previous transaction correctly. >> It doesn't has transaction of 49749 in its backup roots. >> >> Not 100% sure, but looks somewhat like the problem fixed by this patch: >> Btrfs: fix race leading to fs corruption after transaction abortion >> >> It should get backported to all stable release recently. >> >> Thanks, >> Qu >> >> > >> > >> > On 08/15/2019 22:07 +0800, Qu Wenruo wrote: >> >>> >> >>> >> >>> On 2019/8/15 ??????9:52, Tim Walberg wrote: >> >>> > Had to wait for 'btrfs recover' to finish before I proceed farther. >> >>> > >> >>> > Kernel is 4.19.45, tools are 4.19.1 >> >>> > >> >>> > File system is a 3-disk RAID10 with WD3003FZEX (WD Black 3TB) >> >>> > >> >>> > Output from attempting to mount: >> >>> > >> >>> > # mount -o ro,usebackuproot /dev/sdc1 /mnt >> >>> > mount: wrong fs type, bad option, bad superblock on /dev/sdc1, >> >>> > missing codepage or helper program, or other error >> >>> > >> >>> > In some cases useful info is found in syslog - try >> >>> > dmesg | tail or so. >> >>> > >> >>> > Kernel messages from the mount attempt: >> >>> > >> >>> > [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): trying to use backup root at mount time >> >>> > [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): disk space caching is enabled >> >>> > [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): has skinny extents >> >>> > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750 >> >>> > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750 >> >>> > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): failed to read block groups: -5 >> >>> >> >>> Extent tree corruption. >> >>> >> >>> So if that's the only corruption, you have a very high chance to recover >> >>> most of your data. >> >>> >> >>> Btrfs rescue can work, or you can try the experimental patches which >> >>> provides rescue=skip_bg mount option to allow you mount the fs RO and >> >>> receive your data (later is way faster than user space rescue) >> >>> https://patchwork.kernel.org/project/linux-btrfs/list/?series=130637 >> >>> >> >>> Also, for your dump super output, it doesn't provide too much info. >> >>> >> >>> You would like to use -Ffa option for more info. >> >>> Also, you could also try that on all 3 devices, to find out which one >> >>> has lower generation. >> >>> >> >>> Also, please provide the history of the corruption. >> >>> One generation corruptions is a little rare. Is sudden power loss >> >>> involved in this case? >> >>> >> >>> Thanks, >> >>> Qu >> >>> >> >>> > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): open_ctree failed >> >>> > >> >>> > Output from 'btrfs check -p /dev/sdc1': >> >>> > >> >>> > # btrfs check -p /dev/sdc1 >> >>> > Opening filesystem to check... >> >>> > parent transid verify failed on 229846466560 wanted 49749 found 49750 >> >>> > Ignoring transid failure >> >>> > ERROR: child eb corrupted: parent bytenr=229845336064 item=0 parent level=1 child level=2 >> >>> > ERROR: cannot open file system >> >>> > >> >>> > >> >>> > >> >>> > On 08/15/2019 10:35 +0800, Qu Wenruo wrote: >> >>> >>> >> >>> >>> >> >>> >>> On 2019/8/15 ??????2:32, Tim Walberg wrote: >> >>> >>> > Most of the recommendations I've found online deal with when "wanted" is >> >>> >>> > greater than "found", which, if I understand correctly means that one or >> >>> >>> > more transactions were interrupted/lost before fully committed. >> >>> >>> >> >>> >>> No matter what the case is, a proper transaction shouldn't have any tree >> >>> >>> block overwritten. >> >>> >>> >> >>> >>> That means, either the FLUSH/FUA of the hardware/lower block layer is >> >>> >>> screwed up, or the COW of tree block is already screwed up. >> >>> >>> >> >>> >>> > >> >>> >>> > Are the recommendations for recovery the same if the system is reporting a >> >>> >>> > "wanted" that is less than "found"? >> >>> >>> > >> >>> >>> The salvage is no difference than any transid mismatch, no matter if >> >>> >>> it's larger or smaller. >> >>> >>> >> >>> >>> It depends on the tree block. >> >>> >>> >> >>> >>> Please provide full dmesg output and btrfs check for further advice. >> >>> >>> >> >>> >>> Thanks, >> >>> >>> Qu >> >>> >>> >> >>> > >> >>> > >> >>> > >> >>> > >> >>> >> > >> > >> > >> > End of included message >> > >> > >> > >> End of included message -- +----------------------+ | Tim Walberg | | 830 Carriage Dr. | | Algonquin, IL 60102 | | twalberg@comcast.net | +----------------------+ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: recovering from "parent transid verify failed" 2019-08-15 2:35 ` Qu Wenruo 2019-08-15 13:52 ` Tim Walberg @ 2019-08-15 13:55 ` Tim Walberg 1 sibling, 0 replies; 8+ messages in thread From: Tim Walberg @ 2019-08-15 13:55 UTC (permalink / raw) To: Qu Wenruo; +Cc: Tim Walberg, linux-btrfs Also - here's 'btrfs inspect-internal dump-super /dev/sdc1': superblock: bytenr=65536, device=/dev/sdc1 --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0x4331039b [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 53749823-faaf-46f9-866d-3778d93cb1ca label btrfs1 generation 49750 root 229846646784 sys_array_size 129 chunk_root_generation 49725 root_level 1 chunk_root 2568857059328 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 9001775738880 bytes_used 2085801975808 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 3 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x361 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA | NO_HOLES ) cache_generation 49748 uuid_tree_generation 49748 dev_item.uuid 7338a973-9a45-4032-a4c9-d18142fd7908 dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match] dev_item.type 0 dev_item.total_bytes 3000591912960 dev_item.bytes_used 1407675531264 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 Although, 'btrfs inspect-internal logical-resolve ...' just says: # btrfs inspect-internal logical-resolve 229846466560 /dev/sdc1 ERROR: not a btrfs filesystem: /dev/sdc1 On 08/15/2019 10:35 +0800, Qu Wenruo wrote: >> >> >> On 2019/8/15 ??????2:32, Tim Walberg wrote: >> > Most of the recommendations I've found online deal with when "wanted" is >> > greater than "found", which, if I understand correctly means that one or >> > more transactions were interrupted/lost before fully committed. >> >> No matter what the case is, a proper transaction shouldn't have any tree >> block overwritten. >> >> That means, either the FLUSH/FUA of the hardware/lower block layer is >> screwed up, or the COW of tree block is already screwed up. >> >> > >> > Are the recommendations for recovery the same if the system is reporting a >> > "wanted" that is less than "found"? >> > >> The salvage is no difference than any transid mismatch, no matter if >> it's larger or smaller. >> >> It depends on the tree block. >> >> Please provide full dmesg output and btrfs check for further advice. >> >> Thanks, >> Qu >> End of included message -- +----------------------+ | Tim Walberg | | 830 Carriage Dr. | | Algonquin, IL 60102 | | twalberg@comcast.net | +----------------------+ ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-08-15 15:01 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-08-14 18:32 recovering from "parent transid verify failed" Tim Walberg 2019-08-15 2:35 ` Qu Wenruo 2019-08-15 13:52 ` Tim Walberg 2019-08-15 14:07 ` Qu Wenruo 2019-08-15 14:21 ` Tim Walberg 2019-08-15 14:45 ` Qu Wenruo 2019-08-15 15:01 ` Tim Walberg 2019-08-15 13:55 ` Tim Walberg
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).