* Recovering Btrfs from a freak failure of the disk controller
@ 2021-02-14 20:25 Neal Gompa
2021-02-14 21:33 ` Chris Murphy
` (3 more replies)
0 siblings, 4 replies; 41+ messages in thread
From: Neal Gompa @ 2021-02-14 20:25 UTC (permalink / raw)
To: Btrfs BTRFS
Hey all,
So one of my main computers recently had a disk controller failure
that caused my machine to freeze. After rebooting, Btrfs refuses to
mount. I tried to do a mount and the following errors show up in the
journal:
> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled
> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents
> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895]
> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected
> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895]
> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected
> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root
> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed
I've tried to do -o recovery,ro mount and get the same issue. I can't
seem to find any reasonably good information on how to do recovery in
this scenario, even to just recover enough to copy data off.
I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and
the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm
using btrfs-progs v5.10.
Can anyone help?
--
真実はいつも一つ！/ Always, there's only one truth!
^ permalink raw reply [flat|nested] 41+ messages in thread* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-14 20:25 Recovering Btrfs from a freak failure of the disk controller Neal Gompa @ 2021-02-14 21:33 ` Chris Murphy 2021-02-14 22:04 ` Chris Murphy ` (2 subsequent siblings) 3 siblings, 0 replies; 41+ messages in thread From: Chris Murphy @ 2021-02-14 21:33 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On Sun, Feb 14, 2021 at 1:29 PM Neal Gompa <ngompa13@gmail.com> wrote: > > Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > > Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > > Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > > Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > > Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > > Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > > Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > > Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > > I've tried to do -o recovery,ro mount and get the same issue. I can't > seem to find any reasonably good information on how to do recovery in > this scenario, even to just recover enough to copy data off. > > I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > using btrfs-progs v5.10. > > Can anyone help? >has 888896 expect [0, 888895] Off by one error. I haven't previously seen this with 'invalid inode transid'. There's an old kernel bug (long since fixed) that can inject garbage into the inode transid but that's not what's going on here. What do you get for: btrfs check --readonly In the meantime, it might be worth trying 5.11-rc7 or rc8 with the new 'ro,rescue=all' mount option and see if it can skip over this kind of problem. The "parent transid verify failed" are pretty serious, again not the same thing here. But I'm not sure how resilient repair is for either off by one errors, or bitflips still. -- Chris Murphy ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-14 20:25 Recovering Btrfs from a freak failure of the disk controller Neal Gompa 2021-02-14 21:33 ` Chris Murphy @ 2021-02-14 22:04 ` Chris Murphy 2021-02-14 22:11 ` Chris Murphy 2021-02-16 15:19 ` Josef Bacik 3 siblings, 0 replies; 41+ messages in thread From: Chris Murphy @ 2021-02-14 22:04 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS Can you also include: btrfs insp dump-s I wonder if log replay is indicated by non-zero value for log_root in the super block. If so, you check if: ro,nologreplay or ro,nologreplay,usebackuproot work. -- Chris Murphy ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-14 20:25 Recovering Btrfs from a freak failure of the disk controller Neal Gompa 2021-02-14 21:33 ` Chris Murphy 2021-02-14 22:04 ` Chris Murphy @ 2021-02-14 22:11 ` Chris Murphy 2021-02-14 23:23 ` Neal Gompa 2021-02-16 15:19 ` Josef Bacik 3 siblings, 1 reply; 41+ messages in thread From: Chris Murphy @ 2021-02-14 22:11 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On Sun, Feb 14, 2021 at 1:29 PM Neal Gompa <ngompa13@gmail.com> wrote: > > Hey all, > > So one of my main computers recently had a disk controller failure > that caused my machine to freeze. After rebooting, Btrfs refuses to > mount. I tried to do a mount and the following errors show up in the > journal: > > > Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > > Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > > Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > > Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > > Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > > Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > > Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > > Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > > I've tried to do -o recovery,ro mount and get the same issue. I can't > seem to find any reasonably good information on how to do recovery in > this scenario, even to just recover enough to copy data off. > > I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > using btrfs-progs v5.10. Oh and also that block: btrfs insp dump-t -b 796082176 /dev/sda3 -- Chris Murphy ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-14 22:11 ` Chris Murphy @ 2021-02-14 23:23 ` Neal Gompa 2021-02-14 23:51 ` Chris Murphy 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-02-14 23:23 UTC (permalink / raw) To: Chris Murphy; +Cc: Btrfs BTRFS [-- Attachment #1: Type: text/plain, Size: 2759 bytes --] On Sun, Feb 14, 2021 at 5:11 PM Chris Murphy <lists@colorremedies.com> wrote: > > On Sun, Feb 14, 2021 at 1:29 PM Neal Gompa <ngompa13@gmail.com> wrote: > > > > Hey all, > > > > So one of my main computers recently had a disk controller failure > > that caused my machine to freeze. After rebooting, Btrfs refuses to > > mount. I tried to do a mount and the following errors show up in the > > journal: > > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > > > Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > > > Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > > > Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > > > Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > > > Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > > > Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > > > Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > > > > I've tried to do -o recovery,ro mount and get the same issue. I can't > > seem to find any reasonably good information on how to do recovery in > > this scenario, even to just recover enough to copy data off. > > > > I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > > the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > > using btrfs-progs v5.10. > > Oh and also that block: > > btrfs insp dump-t -b 796082176 /dev/sda3 > So, I've attached the output of the dump-s and dump-t commands. As for the other commands: # btrfs check --readonly /dev/sda3 > Opening filesystem to check... > parent transid verify failed on 796082176 wanted 888894 found 888896 > parent transid verify failed on 796082176 wanted 888894 found 888896 > parent transid verify failed on 796082176 wanted 888894 found 888896 > Ignoring transid failure > ERROR: could not setup extent tree > ERROR: cannot open file system # mount -o ro,rescue=all /dev/sda3 /mnt > mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sda3, missing codepage or helper program, or other error. This is using the Fedora 34 KDE 20210214 live ISO snapshot, with kernel version 5.11.0-0.rc7.149.fc34.x86_64. -- 真実はいつも一つ！/ Always, there's only one truth! [-- Attachment #2: dump-s --] [-- Type: application/octet-stream, Size: 1274 bytes --] superblock: bytenr=65536, device=/dev/sda3 --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0xa74fc183 [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid f993ffa4-8801-4d57-a087-1c35fd6ece00 metadata_uuid f993ffa4-8801-4d57-a087-1c35fd6ece00 label fedora_f32-ryo-ohki-winvm generation 888894 root 796082176 sys_array_size 129 chunk_root_generation 787284 root_level 1 chunk_root 22036480 chunk_root_level 0 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 263132807168 bytes_used 114928676864 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 1 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x161 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA ) cache_generation 888894 uuid_tree_generation 888894 dev_item.uuid bf50f7f1-bb98-4932-8388-935e4838cb63 dev_item.fsid f993ffa4-8801-4d57-a087-1c35fd6ece00 [match] dev_item.type 0 dev_item.total_bytes 263132807168 dev_item.bytes_used 131021668352 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 [-- Attachment #3: dump-t-block --] [-- Type: application/octet-stream, Size: 23884 bytes --] btrfs-progs v5.10 leaf 796082176 items 97 free space 90 generation 888896 owner 401 leaf 796082176 flags 0x1(WRITTEN) backref revision 1 fs uuid f993ffa4-8801-4d57-a087-1c35fd6ece00 chunk uuid 7eff154b-3550-427e-98cb-7300b3d69ab3 item 0 key (203653 INODE_REF 35455) itemoff 16215 itemsize 68 index 44 namelen 27 name: thread.cpython-39.opt-1.pyc index 46 namelen 21 name: thread.cpython-39.pyc item 1 key (203653 XATTR_ITEM 3817753667) itemoff 16142 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 2 key (203653 EXTENT_DATA 0) itemoff 16089 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5792862208 nr 8192 extent data offset 0 nr 8192 ram 8192 extent compression 0 (none) item 3 key (203654 INODE_ITEM 0) itemoff 15929 itemsize 160 generation 758017 transid 888891 size 16434 nbytes 20480 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 10 flags 0x0(none) atime 1613320571.46819308 (2021-02-14 11:36:11) ctime 1609555202.983976862 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.982976847 (2021-01-01 21:40:02) item 4 key (203654 INODE_REF 35463) itemoff 15857 itemsize 72 index 35 namelen 29 name: __init__.cpython-39.opt-1.pyc index 37 namelen 23 name: __init__.cpython-39.pyc item 5 key (203654 XATTR_ITEM 3817753667) itemoff 15784 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 6 key (203654 EXTENT_DATA 0) itemoff 15731 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5792870400 nr 20480 extent data offset 0 nr 20480 ram 20480 extent compression 0 (none) item 7 key (203655 INODE_ITEM 0) itemoff 15571 itemsize 160 generation 758017 transid 758017 size 9836 nbytes 12288 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 10 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555202.984976878 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.983976862 (2021-01-01 21:40:02) item 8 key (203655 INODE_REF 35463) itemoff 15507 itemsize 64 index 39 namelen 25 name: _aix.cpython-39.opt-1.pyc index 41 namelen 19 name: _aix.cpython-39.pyc item 9 key (203655 XATTR_ITEM 3817753667) itemoff 15434 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 10 key (203655 EXTENT_DATA 0) itemoff 15381 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5792890880 nr 12288 extent data offset 0 nr 12288 ram 12288 extent compression 0 (none) item 11 key (203656 INODE_ITEM 0) itemoff 15221 itemsize 160 generation 758017 transid 888891 size 1931 nbytes 1931 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 10 flags 0x0(none) atime 1613320571.49819328 (2021-02-14 11:36:11) ctime 1609555202.984976878 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.984976878 (2021-01-01 21:40:02) item 12 key (203656 INODE_REF 35463) itemoff 15151 itemsize 70 index 43 namelen 28 name: _endian.cpython-39.opt-1.pyc index 45 namelen 22 name: _endian.cpython-39.pyc item 13 key (203656 XATTR_ITEM 3817753667) itemoff 15078 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 14 key (203656 EXTENT_DATA 0) itemoff 13126 itemsize 1952 generation 758017 type 0 (inline) inline extent data size 1931 ram_bytes 1931 compression 0 (none) item 15 key (203657 INODE_ITEM 0) itemoff 12966 itemsize 160 generation 758017 transid 888896 size 8263 nbytes 12288 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 10 flags 0x0(none) atime 1613320659.206392941 (2021-02-14 11:37:39) ctime 1609555202.986976908 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.984976878 (2021-01-01 21:40:02) item 16 key (203657 INODE_REF 35463) itemoff 12902 itemsize 64 index 47 namelen 25 name: util.cpython-39.opt-1.pyc index 49 namelen 19 name: util.cpython-39.pyc item 17 key (203657 XATTR_ITEM 3817753667) itemoff 12829 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 18 key (203657 EXTENT_DATA 0) itemoff 12776 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5792903168 nr 12288 extent data offset 0 nr 12288 ram 12288 extent compression 0 (none) item 19 key (203658 INODE_ITEM 0) itemoff 12616 itemsize 160 generation 758017 transid 758017 size 5105 nbytes 8192 block group 0 mode 100644 links 3 uid 0 gid 0 rdev 0 sequence 25 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555202.986976908 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.986976908 (2021-01-01 21:40:02) item 20 key (203658 INODE_REF 35463) itemoff 12505 itemsize 111 index 51 namelen 29 name: wintypes.cpython-39.opt-1.pyc index 53 namelen 29 name: wintypes.cpython-39.opt-2.pyc index 55 namelen 23 name: wintypes.cpython-39.pyc item 21 key (203658 XATTR_ITEM 3817753667) itemoff 12432 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 22 key (203658 EXTENT_DATA 0) itemoff 12379 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5792915456 nr 8192 extent data offset 0 nr 8192 ram 8192 extent compression 0 (none) item 23 key (203659 INODE_ITEM 0) itemoff 12219 itemsize 160 generation 758017 transid 758017 size 301 nbytes 301 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 10 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555202.987976923 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.986976908 (2021-01-01 21:40:02) item 24 key (203659 INODE_REF 35478) itemoff 12147 itemsize 72 index 46 namelen 29 name: __init__.cpython-39.opt-1.pyc index 48 namelen 23 name: __init__.cpython-39.pyc item 25 key (203659 XATTR_ITEM 3817753667) itemoff 12074 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 26 key (203659 EXTENT_DATA 0) itemoff 11752 itemsize 322 generation 758017 type 0 (inline) inline extent data size 301 ram_bytes 301 compression 0 (none) item 27 key (203660 INODE_ITEM 0) itemoff 11592 itemsize 160 generation 758017 transid 888591 size 1861 nbytes 1861 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 10 flags 0x0(none) atime 1613276025.688375961 (2021-02-13 23:13:45) ctime 1609555202.988976939 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.987976923 (2021-01-01 21:40:02) item 28 key (203660 INODE_REF 35485) itemoff 11520 itemsize 72 index 35 namelen 29 name: __init__.cpython-39.opt-1.pyc index 37 namelen 23 name: __init__.cpython-39.pyc item 29 key (203660 XATTR_ITEM 3817753667) itemoff 11447 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 30 key (203660 EXTENT_DATA 0) itemoff 9565 itemsize 1882 generation 758017 type 0 (inline) inline extent data size 1861 ram_bytes 1861 compression 0 (none) item 31 key (203661 INODE_ITEM 0) itemoff 9405 itemsize 160 generation 758017 transid 758017 size 3833 nbytes 4096 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 10 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555202.989976954 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.988976939 (2021-01-01 21:40:02) item 32 key (203661 INODE_REF 35485) itemoff 9339 itemsize 66 index 39 namelen 26 name: ascii.cpython-39.opt-1.pyc index 41 namelen 20 name: ascii.cpython-39.pyc item 33 key (203661 XATTR_ITEM 3817753667) itemoff 9266 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 34 key (203661 EXTENT_DATA 0) itemoff 9213 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5792923648 nr 4096 extent data offset 0 nr 4096 ram 4096 extent compression 0 (none) item 35 key (203662 INODE_ITEM 0) itemoff 9053 itemsize 160 generation 758017 transid 758017 size 4289 nbytes 8192 block group 0 mode 100644 links 3 uid 0 gid 0 rdev 0 sequence 25 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555202.990976969 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.989976954 (2021-01-01 21:40:02) item 36 key (203662 INODE_REF 35485) itemoff 8945 itemsize 108 index 43 namelen 28 name: has_key.cpython-39.opt-1.pyc index 45 namelen 28 name: has_key.cpython-39.opt-2.pyc index 47 namelen 22 name: has_key.cpython-39.pyc item 37 key (203662 XATTR_ITEM 3817753667) itemoff 8872 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 38 key (203662 EXTENT_DATA 0) itemoff 8819 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5792927744 nr 8192 extent data offset 0 nr 8192 ram 8192 extent compression 0 (none) item 39 key (203663 INODE_ITEM 0) itemoff 8659 itemsize 160 generation 758017 transid 758017 size 225 nbytes 225 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 22 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555202.990976969 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.990976969 (2021-01-01 21:40:02) item 40 key (203663 INODE_REF 35485) itemoff 8593 itemsize 66 index 49 namelen 26 name: panel.cpython-39.opt-1.pyc index 51 namelen 20 name: panel.cpython-39.pyc item 41 key (203663 XATTR_ITEM 3817753667) itemoff 8520 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 42 key (203663 EXTENT_DATA 0) itemoff 8274 itemsize 246 generation 758017 type 0 (inline) inline extent data size 225 ram_bytes 225 compression 0 (none) item 43 key (203664 INODE_ITEM 0) itemoff 8114 itemsize 160 generation 758017 transid 758017 size 5907 nbytes 8192 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 10 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555202.992977000 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.990976969 (2021-01-01 21:40:02) item 44 key (203664 INODE_REF 35485) itemoff 8044 itemsize 70 index 53 namelen 28 name: textpad.cpython-39.opt-1.pyc index 55 namelen 22 name: textpad.cpython-39.pyc item 45 key (203664 XATTR_ITEM 3817753667) itemoff 7971 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 46 key (203664 EXTENT_DATA 0) itemoff 7918 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5792935936 nr 8192 extent data offset 0 nr 8192 ram 8192 extent compression 0 (none) item 47 key (203665 INODE_ITEM 0) itemoff 7758 itemsize 160 generation 758017 transid 758017 size 4233 nbytes 8192 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 10 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555202.993977015 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.992977000 (2021-01-01 21:40:02) item 48 key (203665 INODE_REF 35486) itemoff 7686 itemsize 72 index 31 namelen 29 name: __init__.cpython-39.opt-1.pyc index 33 namelen 23 name: __init__.cpython-39.pyc item 49 key (203665 XATTR_ITEM 3817753667) itemoff 7613 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 50 key (203665 EXTENT_DATA 0) itemoff 7560 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5792944128 nr 8192 extent data offset 0 nr 8192 ram 8192 extent compression 0 (none) item 51 key (203666 INODE_ITEM 0) itemoff 7400 itemsize 160 generation 758017 transid 758017 size 7915 nbytes 8192 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 22 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555202.993977015 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.993977015 (2021-01-01 21:40:02) item 52 key (203666 INODE_REF 35486) itemoff 7336 itemsize 64 index 35 namelen 25 name: dumb.cpython-39.opt-1.pyc index 37 namelen 19 name: dumb.cpython-39.pyc item 53 key (203666 XATTR_ITEM 3817753667) itemoff 7263 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 54 key (203666 EXTENT_DATA 0) itemoff 7210 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5792952320 nr 8192 extent data offset 0 nr 8192 ram 8192 extent compression 0 (none) item 55 key (203667 INODE_ITEM 0) itemoff 7050 itemsize 160 generation 758017 transid 758017 size 205 nbytes 205 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 22 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555202.994977030 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.993977015 (2021-01-01 21:40:02) item 56 key (203667 INODE_REF 35486) itemoff 6988 itemsize 62 index 39 namelen 24 name: gnu.cpython-39.opt-1.pyc index 41 namelen 18 name: gnu.cpython-39.pyc item 57 key (203667 XATTR_ITEM 3817753667) itemoff 6915 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 58 key (203667 EXTENT_DATA 0) itemoff 6689 itemsize 226 generation 758017 type 0 (inline) inline extent data size 205 ram_bytes 205 compression 0 (none) item 59 key (203668 INODE_ITEM 0) itemoff 6529 itemsize 160 generation 758017 transid 758017 size 204 nbytes 204 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 22 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555202.996977061 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.994977030 (2021-01-01 21:40:02) item 60 key (203668 INODE_REF 35486) itemoff 6465 itemsize 64 index 43 namelen 25 name: ndbm.cpython-39.opt-1.pyc index 45 namelen 19 name: ndbm.cpython-39.pyc item 61 key (203668 XATTR_ITEM 3817753667) itemoff 6392 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 62 key (203668 EXTENT_DATA 0) itemoff 6167 itemsize 225 generation 758017 type 0 (inline) inline extent data size 204 ram_bytes 204 compression 0 (none) item 63 key (203669 INODE_ITEM 0) itemoff 6007 itemsize 160 generation 758017 transid 888634 size 386 nbytes 386 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 22 flags 0x0(none) atime 1613283105.362309913 (2021-02-14 01:11:45) ctime 1609555202.996977061 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.996977061 (2021-01-01 21:40:02) item 64 key (203669 INODE_REF 35487) itemoff 5935 itemsize 72 index 231 namelen 29 name: __init__.cpython-39.opt-1.pyc index 233 namelen 23 name: __init__.cpython-39.pyc item 65 key (203669 XATTR_ITEM 3817753667) itemoff 5862 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 66 key (203669 EXTENT_DATA 0) itemoff 5455 itemsize 407 generation 758017 type 0 (inline) inline extent data size 386 ram_bytes 386 compression 0 (none) item 67 key (203670 INODE_ITEM 0) itemoff 5295 itemsize 160 generation 758017 transid 758017 size 6587 nbytes 8192 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 22 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555202.997977076 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.996977061 (2021-01-01 21:40:02) item 68 key (203670 INODE_REF 35487) itemoff 5215 itemsize 80 index 235 namelen 33 name: archive_util.cpython-39.opt-1.pyc index 237 namelen 27 name: archive_util.cpython-39.pyc item 69 key (203670 XATTR_ITEM 3817753667) itemoff 5142 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 70 key (203670 EXTENT_DATA 0) itemoff 5089 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5792960512 nr 8192 extent data offset 0 nr 8192 ram 8192 extent compression 0 (none) item 71 key (203671 INODE_ITEM 0) itemoff 4929 itemsize 160 generation 758017 transid 758017 size 6498 nbytes 8192 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 22 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555202.998977091 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.997977076 (2021-01-01 21:40:02) item 72 key (203671 INODE_REF 35487) itemoff 4849 itemsize 80 index 239 namelen 33 name: bcppcompiler.cpython-39.opt-1.pyc index 241 namelen 27 name: bcppcompiler.cpython-39.pyc item 73 key (203671 XATTR_ITEM 3817753667) itemoff 4776 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 74 key (203671 EXTENT_DATA 0) itemoff 4723 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5792968704 nr 8192 extent data offset 0 nr 8192 ram 8192 extent compression 0 (none) item 75 key (203672 INODE_ITEM 0) itemoff 4563 itemsize 160 generation 758017 transid 758017 size 13926 nbytes 16384 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 22 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555202.999977106 (2021-01-01 21:40:02) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.998977091 (2021-01-01 21:40:02) item 76 key (203672 INODE_REF 35487) itemoff 4501 itemsize 62 index 243 namelen 24 name: cmd.cpython-39.opt-1.pyc index 245 namelen 18 name: cmd.cpython-39.pyc item 77 key (203672 XATTR_ITEM 3817753667) itemoff 4428 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 78 key (203672 EXTENT_DATA 0) itemoff 4375 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5792976896 nr 16384 extent data offset 0 nr 16384 ram 16384 extent compression 0 (none) item 79 key (203673 INODE_ITEM 0) itemoff 4215 itemsize 160 generation 758017 transid 758017 size 3529 nbytes 4096 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 22 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555203.977121 (2021-01-01 21:40:03) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555202.999977106 (2021-01-01 21:40:02) item 80 key (203673 INODE_REF 35487) itemoff 4147 itemsize 68 index 247 namelen 27 name: config.cpython-39.opt-1.pyc index 249 namelen 21 name: config.cpython-39.pyc item 81 key (203673 XATTR_ITEM 3817753667) itemoff 4074 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 82 key (203673 EXTENT_DATA 0) itemoff 4021 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5792993280 nr 4096 extent data offset 0 nr 4096 ram 4096 extent compression 0 (none) item 83 key (203674 INODE_ITEM 0) itemoff 3861 itemsize 160 generation 758017 transid 758017 size 6654 nbytes 8192 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 25 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555203.2977152 (2021-01-01 21:40:03) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555203.977121 (2021-01-01 21:40:03) item 84 key (203674 INODE_REF 35487) itemoff 3797 itemsize 64 index 251 namelen 25 name: core.cpython-39.opt-1.pyc index 253 namelen 19 name: core.cpython-39.pyc item 85 key (203674 XATTR_ITEM 3817753667) itemoff 3724 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 86 key (203674 EXTENT_DATA 0) itemoff 3671 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5792997376 nr 8192 extent data offset 0 nr 8192 ram 8192 extent compression 0 (none) item 87 key (203675 INODE_ITEM 0) itemoff 3511 itemsize 160 generation 758017 transid 758017 size 8503 nbytes 12288 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 10 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555203.2977152 (2021-01-01 21:40:03) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555203.2977152 (2021-01-01 21:40:03) item 88 key (203675 INODE_REF 35487) itemoff 3425 itemsize 86 index 255 namelen 36 name: cygwinccompiler.cpython-39.opt-1.pyc index 257 namelen 30 name: cygwinccompiler.cpython-39.pyc item 89 key (203675 XATTR_ITEM 3817753667) itemoff 3352 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 90 key (203675 EXTENT_DATA 0) itemoff 3299 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5793005568 nr 12288 extent data offset 0 nr 12288 ram 12288 extent compression 0 (none) item 91 key (203676 INODE_ITEM 0) itemoff 3139 itemsize 160 generation 758017 transid 888029 size 196 nbytes 196 block group 0 mode 100644 links 3 uid 0 gid 0 rdev 0 sequence 13 flags 0x0(none) atime 1613236757.512324144 (2021-02-13 12:19:17) ctime 1609555203.3977167 (2021-01-01 21:40:03) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555203.2977152 (2021-01-01 21:40:03) item 92 key (203676 INODE_REF 35487) itemoff 3037 itemsize 102 index 259 namelen 26 name: debug.cpython-39.opt-1.pyc index 261 namelen 26 name: debug.cpython-39.opt-2.pyc index 263 namelen 20 name: debug.cpython-39.pyc item 93 key (203676 XATTR_ITEM 3817753667) itemoff 2964 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 94 key (203676 EXTENT_DATA 0) itemoff 2747 itemsize 217 generation 758017 type 0 (inline) inline extent data size 196 ram_bytes 196 compression 0 (none) item 95 key (203677 INODE_ITEM 0) itemoff 2587 itemsize 160 generation 758017 transid 888896 size 2716 nbytes 4096 block group 0 mode 100644 links 2 uid 0 gid 0 rdev 0 sequence 10 flags 0x0(none) atime 1613320654.132359925 (2021-02-14 11:37:34) ctime 1609555203.5977198 (2021-01-01 21:40:03) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555203.3977167 (2021-01-01 21:40:03) item 96 key (203677 INODE_REF 35487) itemoff 2515 itemsize 72 index 265 namelen 29 name: dep_util.cpython-39.opt-1.pyc index 267 namelen 23 name: dep_util.cpython-39.pyc ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-14 23:23 ` Neal Gompa @ 2021-02-14 23:51 ` Chris Murphy 2021-02-14 23:56 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Chris Murphy @ 2021-02-14 23:51 UTC (permalink / raw) To: Neal Gompa; +Cc: Chris Murphy, Btrfs BTRFS On Sun, Feb 14, 2021 at 4:24 PM Neal Gompa <ngompa13@gmail.com> wrote: > > On Sun, Feb 14, 2021 at 5:11 PM Chris Murphy <lists@colorremedies.com> wrote: > > > > On Sun, Feb 14, 2021 at 1:29 PM Neal Gompa <ngompa13@gmail.com> wrote: > > > > > > Hey all, > > > > > > So one of my main computers recently had a disk controller failure > > > that caused my machine to freeze. After rebooting, Btrfs refuses to > > > mount. I tried to do a mount and the following errors show up in the > > > journal: > > > > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > > > > > > I've tried to do -o recovery,ro mount and get the same issue. I can't > > > seem to find any reasonably good information on how to do recovery in > > > this scenario, even to just recover enough to copy data off. > > > > > > I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > > > the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > > > using btrfs-progs v5.10. > > > > Oh and also that block: > > > > btrfs insp dump-t -b 796082176 /dev/sda3 > > > > So, I've attached the output of the dump-s and dump-t commands. > > As for the other commands: > > # btrfs check --readonly /dev/sda3 > > Opening filesystem to check... > > parent transid verify failed on 796082176 wanted 888894 found 888896 Not good. So three different transids in play. Super says generation 888894 Leaf block says its generation is 888896, and two inodes have transid 888896 including the one the tree checker is complaining about. Somehow the super has an older generation than both what's in the leaf and what's expected. > > parent transid verify failed on 796082176 wanted 888894 found 888896 > > parent transid verify failed on 796082176 wanted 888894 found 888896 > > Ignoring transid failure > > ERROR: could not setup extent tree > > ERROR: cannot open file system > > # mount -o ro,rescue=all /dev/sda3 /mnt > > mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sda3, missing codepage or helper program, or other error. Do you get the same kernel messages as originally reported? Or something different? -- Chris Murphy ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-14 23:51 ` Chris Murphy @ 2021-02-14 23:56 ` Neal Gompa 0 siblings, 0 replies; 41+ messages in thread From: Neal Gompa @ 2021-02-14 23:56 UTC (permalink / raw) To: Chris Murphy; +Cc: Btrfs BTRFS On Sun, Feb 14, 2021 at 6:52 PM Chris Murphy <lists@colorremedies.com> wrote: > > On Sun, Feb 14, 2021 at 4:24 PM Neal Gompa <ngompa13@gmail.com> wrote: > > > > On Sun, Feb 14, 2021 at 5:11 PM Chris Murphy <lists@colorremedies.com> wrote: > > > > > > On Sun, Feb 14, 2021 at 1:29 PM Neal Gompa <ngompa13@gmail.com> wrote: > > > > > > > > Hey all, > > > > > > > > So one of my main computers recently had a disk controller failure > > > > that caused my machine to freeze. After rebooting, Btrfs refuses to > > > > mount. I tried to do a mount and the following errors show up in the > > > > journal: > > > > > > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > > > > > Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > > > > > > > > I've tried to do -o recovery,ro mount and get the same issue. I can't > > > > seem to find any reasonably good information on how to do recovery in > > > > this scenario, even to just recover enough to copy data off. > > > > > > > > I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > > > > the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > > > > using btrfs-progs v5.10. > > > > > > Oh and also that block: > > > > > > btrfs insp dump-t -b 796082176 /dev/sda3 > > > > > > > So, I've attached the output of the dump-s and dump-t commands. > > > > As for the other commands: > > > > # btrfs check --readonly /dev/sda3 > > > Opening filesystem to check... > > > parent transid verify failed on 796082176 wanted 888894 found 888896 > > Not good. So three different transids in play. > > Super says generation 888894 > > Leaf block says its generation is 888896, and two inodes have transid > 888896 including the one the tree checker is complaining about. > Somehow the super has an older generation than both what's in the leaf > and what's expected. > Uhh, that's a lot of oddities there... > > > parent transid verify failed on 796082176 wanted 888894 found 888896 > > > parent transid verify failed on 796082176 wanted 888894 found 888896 > > > Ignoring transid failure > > > ERROR: could not setup extent tree > > > ERROR: cannot open file system > > > > # mount -o ro,rescue=all /dev/sda3 /mnt > > > mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sda3, missing codepage or helper program, or other error. > > Do you get the same kernel messages as originally reported? Or > something different? > Not substantially different: > Feb 14 18:54:06 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options > Feb 14 18:54:06 localhost-live kernel: BTRFS info (device sda3): ignoring data csums > Feb 14 18:54:06 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots > Feb 14 18:54:06 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time > Feb 14 18:54:06 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > Feb 14 18:54:06 localhost-live kernel: BTRFS info (device sda3): has skinny extents > Feb 14 18:54:06 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > Feb 14 18:54:06 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > Feb 14 18:54:06 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > Feb 14 18:54:06 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > Feb 14 18:54:06 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > Feb 14 18:54:06 localhost-live kernel: BTRFS error (device sda3): open_ctree failed -- 真実はいつも一つ！/ Always, there's only one truth! ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-14 20:25 Recovering Btrfs from a freak failure of the disk controller Neal Gompa ` (2 preceding siblings ...) 2021-02-14 22:11 ` Chris Murphy @ 2021-02-16 15:19 ` Josef Bacik 2021-02-16 16:27 ` Neal Gompa 3 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-02-16 15:19 UTC (permalink / raw) To: Neal Gompa, Btrfs BTRFS On 2/14/21 3:25 PM, Neal Gompa wrote: > Hey all, > > So one of my main computers recently had a disk controller failure > that caused my machine to freeze. After rebooting, Btrfs refuses to > mount. I tried to do a mount and the following errors show up in the > journal: > >> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > > I've tried to do -o recovery,ro mount and get the same issue. I can't > seem to find any reasonably good information on how to do recovery in > this scenario, even to just recover enough to copy data off. > > I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > using btrfs-progs v5.10. > > Can anyone help? Can you try btrfs check --clear-space-cache v1 /dev/whatever That should fix the inode generation thing so it's sane, and then the tree checker will allow the fs to be read, hopefully. If not we can work out some other magic. Thanks, Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-16 15:19 ` Josef Bacik @ 2021-02-16 16:27 ` Neal Gompa 2021-02-16 18:11 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-02-16 16:27 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > > On 2/14/21 3:25 PM, Neal Gompa wrote: > > Hey all, > > > > So one of my main computers recently had a disk controller failure > > that caused my machine to freeze. After rebooting, Btrfs refuses to > > mount. I tried to do a mount and the following errors show up in the > > journal: > > > >> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > > > > I've tried to do -o recovery,ro mount and get the same issue. I can't > > seem to find any reasonably good information on how to do recovery in > > this scenario, even to just recover enough to copy data off. > > > > I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > > the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > > using btrfs-progs v5.10. > > > > Can anyone help? > > Can you try > > btrfs check --clear-space-cache v1 /dev/whatever > > That should fix the inode generation thing so it's sane, and then the tree > checker will allow the fs to be read, hopefully. If not we can work out some > other magic. Thanks, > > Josef I got the same error as I did with btrfs-check --readonly... # btrfs check --clear-space-cache v1 /dev/sda3 > Opening filesystem to check... > parent transid verify failed on 796082176 wanted 888894 found 888896 > parent transid verify failed on 796082176 wanted 888894 found 888896 > parent transid verify failed on 796082176 wanted 888894 found 888896 > Ignoring transid failure > ERROR: could not setup extent tree > ERROR: cannot open file system -- 真実はいつも一つ！/ Always, there's only one truth! ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-16 16:27 ` Neal Gompa @ 2021-02-16 18:11 ` Josef Bacik 2021-02-16 20:29 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-02-16 18:11 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 2/16/21 11:27 AM, Neal Gompa wrote: > On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 2/14/21 3:25 PM, Neal Gompa wrote: >>> Hey all, >>> >>> So one of my main computers recently had a disk controller failure >>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>> mount. I tried to do a mount and the following errors show up in the >>> journal: >>> >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>> >>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>> seem to find any reasonably good information on how to do recovery in >>> this scenario, even to just recover enough to copy data off. >>> >>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>> using btrfs-progs v5.10. >>> >>> Can anyone help? >> >> Can you try >> >> btrfs check --clear-space-cache v1 /dev/whatever >> >> That should fix the inode generation thing so it's sane, and then the tree >> checker will allow the fs to be read, hopefully. If not we can work out some >> other magic. Thanks, >> >> Josef > > I got the same error as I did with btrfs-check --readonly... > Oh lovely, what does btrfs check --readonly --backup do? Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-16 18:11 ` Josef Bacik @ 2021-02-16 20:29 ` Neal Gompa 2021-02-16 21:24 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-02-16 20:29 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > > On 2/16/21 11:27 AM, Neal Gompa wrote: > > On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>> Hey all, > >>> > >>> So one of my main computers recently had a disk controller failure > >>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>> mount. I tried to do a mount and the following errors show up in the > >>> journal: > >>> > >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>> > >>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>> seem to find any reasonably good information on how to do recovery in > >>> this scenario, even to just recover enough to copy data off. > >>> > >>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>> using btrfs-progs v5.10. > >>> > >>> Can anyone help? > >> > >> Can you try > >> > >> btrfs check --clear-space-cache v1 /dev/whatever > >> > >> That should fix the inode generation thing so it's sane, and then the tree > >> checker will allow the fs to be read, hopefully. If not we can work out some > >> other magic. Thanks, > >> > >> Josef > > > > I got the same error as I did with btrfs-check --readonly... > > > > Oh lovely, what does btrfs check --readonly --backup do? > No dice... # btrfs check --readonly --backup /dev/sda3 > Opening filesystem to check... > parent transid verify failed on 791281664 wanted 888893 found 888895 > parent transid verify failed on 791281664 wanted 888893 found 888895 > parent transid verify failed on 791281664 wanted 888893 found 888895 > Ignoring transid failure > ERROR: could not setup extent tree > ERROR: cannot open file system -- 真実はいつも一つ！/ Always, there's only one truth! ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-16 20:29 ` Neal Gompa @ 2021-02-16 21:24 ` Josef Bacik 2021-02-17 2:05 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-02-16 21:24 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 2/16/21 3:29 PM, Neal Gompa wrote: > On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 2/16/21 11:27 AM, Neal Gompa wrote: >>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>> >>>> On 2/14/21 3:25 PM, Neal Gompa wrote: >>>>> Hey all, >>>>> >>>>> So one of my main computers recently had a disk controller failure >>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>>>> mount. I tried to do a mount and the following errors show up in the >>>>> journal: >>>>> >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>> >>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>>>> seem to find any reasonably good information on how to do recovery in >>>>> this scenario, even to just recover enough to copy data off. >>>>> >>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>>>> using btrfs-progs v5.10. >>>>> >>>>> Can anyone help? >>>> >>>> Can you try >>>> >>>> btrfs check --clear-space-cache v1 /dev/whatever >>>> >>>> That should fix the inode generation thing so it's sane, and then the tree >>>> checker will allow the fs to be read, hopefully. If not we can work out some >>>> other magic. Thanks, >>>> >>>> Josef >>> >>> I got the same error as I did with btrfs-check --readonly... >>> >> >> Oh lovely, what does btrfs check --readonly --backup do? >> > > No dice... > > # btrfs check --readonly --backup /dev/sda3 >> Opening filesystem to check... >> parent transid verify failed on 791281664 wanted 888893 found 888895 >> parent transid verify failed on 791281664 wanted 888893 found 888895 >> parent transid verify failed on 791281664 wanted 888893 found 888895 Hey look the block we're looking for, I wrote you some magic, just pull https://github.com/josefbacik/btrfs-progs/tree/for-neal build, and then run btrfs-neal-magic /dev/sda3 791281664 888895 This will force us to point at the old root with (hopefully) the right bytenr and gen, and then hopefully you'll be able to recover from there. This is kind of saucy, so yolo, but I can undo it if it makes things worse. Thanks, Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-16 21:24 ` Josef Bacik @ 2021-02-17 2:05 ` Neal Gompa 2021-02-17 14:36 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-02-17 2:05 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: > > On 2/16/21 3:29 PM, Neal Gompa wrote: > > On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>> > >>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>> Hey all, > >>>>> > >>>>> So one of my main computers recently had a disk controller failure > >>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>>>> mount. I tried to do a mount and the following errors show up in the > >>>>> journal: > >>>>> > >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>> > >>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>>>> seem to find any reasonably good information on how to do recovery in > >>>>> this scenario, even to just recover enough to copy data off. > >>>>> > >>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>>>> using btrfs-progs v5.10. > >>>>> > >>>>> Can anyone help? > >>>> > >>>> Can you try > >>>> > >>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>> > >>>> That should fix the inode generation thing so it's sane, and then the tree > >>>> checker will allow the fs to be read, hopefully. If not we can work out some > >>>> other magic. Thanks, > >>>> > >>>> Josef > >>> > >>> I got the same error as I did with btrfs-check --readonly... > >>> > >> > >> Oh lovely, what does btrfs check --readonly --backup do? > >> > > > > No dice... > > > > # btrfs check --readonly --backup /dev/sda3 > >> Opening filesystem to check... > >> parent transid verify failed on 791281664 wanted 888893 found 888895 > >> parent transid verify failed on 791281664 wanted 888893 found 888895 > >> parent transid verify failed on 791281664 wanted 888893 found 888895 > > Hey look the block we're looking for, I wrote you some magic, just pull > > https://github.com/josefbacik/btrfs-progs/tree/for-neal > > build, and then run > > btrfs-neal-magic /dev/sda3 791281664 888895 > > This will force us to point at the old root with (hopefully) the right bytenr > and gen, and then hopefully you'll be able to recover from there. This is kind > of saucy, so yolo, but I can undo it if it makes things worse. Thanks, > # btrfs check --readonly /dev/sda3 > Opening filesystem to check... > ERROR: could not setup extent tree > ERROR: cannot open file system # btrfs check --clear-space-cache v1 /dev/sda3 > Opening filesystem to check... > ERROR: could not setup extent tree > ERROR: cannot open file system It's better, but still no dice... :( -- 真実はいつも一つ！/ Always, there's only one truth! ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-17 2:05 ` Neal Gompa @ 2021-02-17 14:36 ` Josef Bacik 2021-02-17 14:50 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-02-17 14:36 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 2/16/21 9:05 PM, Neal Gompa wrote: > On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 2/16/21 3:29 PM, Neal Gompa wrote: >>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>> >>>> On 2/16/21 11:27 AM, Neal Gompa wrote: >>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>> >>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: >>>>>>> Hey all, >>>>>>> >>>>>>> So one of my main computers recently had a disk controller failure >>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>>>>>> mount. I tried to do a mount and the following errors show up in the >>>>>>> journal: >>>>>>> >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>> >>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>>>>>> seem to find any reasonably good information on how to do recovery in >>>>>>> this scenario, even to just recover enough to copy data off. >>>>>>> >>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>>>>>> using btrfs-progs v5.10. >>>>>>> >>>>>>> Can anyone help? >>>>>> >>>>>> Can you try >>>>>> >>>>>> btrfs check --clear-space-cache v1 /dev/whatever >>>>>> >>>>>> That should fix the inode generation thing so it's sane, and then the tree >>>>>> checker will allow the fs to be read, hopefully. If not we can work out some >>>>>> other magic. Thanks, >>>>>> >>>>>> Josef >>>>> >>>>> I got the same error as I did with btrfs-check --readonly... >>>>> >>>> >>>> Oh lovely, what does btrfs check --readonly --backup do? >>>> >>> >>> No dice... >>> >>> # btrfs check --readonly --backup /dev/sda3 >>>> Opening filesystem to check... >>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >> >> Hey look the block we're looking for, I wrote you some magic, just pull >> >> https://github.com/josefbacik/btrfs-progs/tree/for-neal >> >> build, and then run >> >> btrfs-neal-magic /dev/sda3 791281664 888895 >> >> This will force us to point at the old root with (hopefully) the right bytenr >> and gen, and then hopefully you'll be able to recover from there. This is kind >> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, >> > > # btrfs check --readonly /dev/sda3 >> Opening filesystem to check... >> ERROR: could not setup extent tree >> ERROR: cannot open file system > # btrfs check --clear-space-cache v1 /dev/sda3 >> Opening filesystem to check... >> ERROR: could not setup extent tree >> ERROR: cannot open file system > > It's better, but still no dice... :( > > Hmm it's not telling us what's wrong with the extent tree, which is annoying. Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-17 14:36 ` Josef Bacik @ 2021-02-17 14:50 ` Neal Gompa 2021-02-17 14:59 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-02-17 14:50 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: > > On 2/16/21 9:05 PM, Neal Gompa wrote: > > On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 2/16/21 3:29 PM, Neal Gompa wrote: > >>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>> > >>>> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>> > >>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>>>> Hey all, > >>>>>>> > >>>>>>> So one of my main computers recently had a disk controller failure > >>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>>>>>> mount. I tried to do a mount and the following errors show up in the > >>>>>>> journal: > >>>>>>> > >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>> > >>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>>>>>> seem to find any reasonably good information on how to do recovery in > >>>>>>> this scenario, even to just recover enough to copy data off. > >>>>>>> > >>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>>>>>> using btrfs-progs v5.10. > >>>>>>> > >>>>>>> Can anyone help? > >>>>>> > >>>>>> Can you try > >>>>>> > >>>>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>>>> > >>>>>> That should fix the inode generation thing so it's sane, and then the tree > >>>>>> checker will allow the fs to be read, hopefully. If not we can work out some > >>>>>> other magic. Thanks, > >>>>>> > >>>>>> Josef > >>>>> > >>>>> I got the same error as I did with btrfs-check --readonly... > >>>>> > >>>> > >>>> Oh lovely, what does btrfs check --readonly --backup do? > >>>> > >>> > >>> No dice... > >>> > >>> # btrfs check --readonly --backup /dev/sda3 > >>>> Opening filesystem to check... > >>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >> > >> Hey look the block we're looking for, I wrote you some magic, just pull > >> > >> https://github.com/josefbacik/btrfs-progs/tree/for-neal > >> > >> build, and then run > >> > >> btrfs-neal-magic /dev/sda3 791281664 888895 > >> > >> This will force us to point at the old root with (hopefully) the right bytenr > >> and gen, and then hopefully you'll be able to recover from there. This is kind > >> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, > >> > > > > # btrfs check --readonly /dev/sda3 > >> Opening filesystem to check... > >> ERROR: could not setup extent tree > >> ERROR: cannot open file system > > # btrfs check --clear-space-cache v1 /dev/sda3 > >> Opening filesystem to check... > >> ERROR: could not setup extent tree > >> ERROR: cannot open file system > > > > It's better, but still no dice... :( > > > > > > Hmm it's not telling us what's wrong with the extent tree, which is annoying. > Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, > Nope, I see this in the journal: > Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options > Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums > Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots > Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time > Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents > Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed -- 真実はいつも一つ！/ Always, there's only one truth! ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-17 14:50 ` Neal Gompa @ 2021-02-17 14:59 ` Josef Bacik 2021-02-17 16:29 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-02-17 14:59 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 2/17/21 9:50 AM, Neal Gompa wrote: > On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 2/16/21 9:05 PM, Neal Gompa wrote: >>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>> >>>> On 2/16/21 3:29 PM, Neal Gompa wrote: >>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>> >>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: >>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>> >>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: >>>>>>>>> Hey all, >>>>>>>>> >>>>>>>>> So one of my main computers recently had a disk controller failure >>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>>>>>>>> mount. I tried to do a mount and the following errors show up in the >>>>>>>>> journal: >>>>>>>>> >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>> >>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>>>>>>>> seem to find any reasonably good information on how to do recovery in >>>>>>>>> this scenario, even to just recover enough to copy data off. >>>>>>>>> >>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>>>>>>>> using btrfs-progs v5.10. >>>>>>>>> >>>>>>>>> Can anyone help? >>>>>>>> >>>>>>>> Can you try >>>>>>>> >>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever >>>>>>>> >>>>>>>> That should fix the inode generation thing so it's sane, and then the tree >>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some >>>>>>>> other magic. Thanks, >>>>>>>> >>>>>>>> Josef >>>>>>> >>>>>>> I got the same error as I did with btrfs-check --readonly... >>>>>>> >>>>>> >>>>>> Oh lovely, what does btrfs check --readonly --backup do? >>>>>> >>>>> >>>>> No dice... >>>>> >>>>> # btrfs check --readonly --backup /dev/sda3 >>>>>> Opening filesystem to check... >>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>> >>>> Hey look the block we're looking for, I wrote you some magic, just pull >>>> >>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal >>>> >>>> build, and then run >>>> >>>> btrfs-neal-magic /dev/sda3 791281664 888895 >>>> >>>> This will force us to point at the old root with (hopefully) the right bytenr >>>> and gen, and then hopefully you'll be able to recover from there. This is kind >>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, >>>> >>> >>> # btrfs check --readonly /dev/sda3 >>>> Opening filesystem to check... >>>> ERROR: could not setup extent tree >>>> ERROR: cannot open file system >>> # btrfs check --clear-space-cache v1 /dev/sda3 >>>> Opening filesystem to check... >>>> ERROR: could not setup extent tree >>>> ERROR: cannot open file system >>> >>> It's better, but still no dice... :( >>> >>> >> >> Hmm it's not telling us what's wrong with the extent tree, which is annoying. >> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, >> > > Nope, I see this in the journal: > >> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options >> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums >> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots >> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time >> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents >> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > > Ok git pull for-neal, rebuild, then run btrfs-neal-magic /dev/sda3 791281664 888895 2 I thought of this yesterday but in my head was like "naaahhhh, whats the chances that the level doesn't match??". Thanks, Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-17 14:59 ` Josef Bacik @ 2021-02-17 16:29 ` Neal Gompa 2021-02-17 16:44 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-02-17 16:29 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS [-- Attachment #1: Type: text/plain, Size: 6144 bytes --] On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: > > On 2/17/21 9:50 AM, Neal Gompa wrote: > > On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 2/16/21 9:05 PM, Neal Gompa wrote: > >>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>> > >>>> On 2/16/21 3:29 PM, Neal Gompa wrote: > >>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>> > >>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>> > >>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>>>>>> Hey all, > >>>>>>>>> > >>>>>>>>> So one of my main computers recently had a disk controller failure > >>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>>>>>>>> mount. I tried to do a mount and the following errors show up in the > >>>>>>>>> journal: > >>>>>>>>> > >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>> > >>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>>>>>>>> seem to find any reasonably good information on how to do recovery in > >>>>>>>>> this scenario, even to just recover enough to copy data off. > >>>>>>>>> > >>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>>>>>>>> using btrfs-progs v5.10. > >>>>>>>>> > >>>>>>>>> Can anyone help? > >>>>>>>> > >>>>>>>> Can you try > >>>>>>>> > >>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>>>>>> > >>>>>>>> That should fix the inode generation thing so it's sane, and then the tree > >>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some > >>>>>>>> other magic. Thanks, > >>>>>>>> > >>>>>>>> Josef > >>>>>>> > >>>>>>> I got the same error as I did with btrfs-check --readonly... > >>>>>>> > >>>>>> > >>>>>> Oh lovely, what does btrfs check --readonly --backup do? > >>>>>> > >>>>> > >>>>> No dice... > >>>>> > >>>>> # btrfs check --readonly --backup /dev/sda3 > >>>>>> Opening filesystem to check... > >>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>> > >>>> Hey look the block we're looking for, I wrote you some magic, just pull > >>>> > >>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal > >>>> > >>>> build, and then run > >>>> > >>>> btrfs-neal-magic /dev/sda3 791281664 888895 > >>>> > >>>> This will force us to point at the old root with (hopefully) the right bytenr > >>>> and gen, and then hopefully you'll be able to recover from there. This is kind > >>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, > >>>> > >>> > >>> # btrfs check --readonly /dev/sda3 > >>>> Opening filesystem to check... > >>>> ERROR: could not setup extent tree > >>>> ERROR: cannot open file system > >>> # btrfs check --clear-space-cache v1 /dev/sda3 > >>>> Opening filesystem to check... > >>>> ERROR: could not setup extent tree > >>>> ERROR: cannot open file system > >>> > >>> It's better, but still no dice... :( > >>> > >>> > >> > >> Hmm it's not telling us what's wrong with the extent tree, which is annoying. > >> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, > >> > > > > Nope, I see this in the journal: > > > >> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options > >> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums > >> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots > >> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time > >> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > > > > > > Ok git pull for-neal, rebuild, then run > > btrfs-neal-magic /dev/sda3 791281664 888895 2 > > I thought of this yesterday but in my head was like "naaahhhh, whats the chances > that the level doesn't match??". Thanks, > Tried rescue mount again after running that and got a stack trace in the kernel, detailed in the following attached log. -- 真実はいつも一つ！/ Always, there's only one truth! [-- Attachment #2: output-clean.log --] [-- Type: text/x-log, Size: 7099 bytes --] Feb 17 11:24:35 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options Feb 17 11:24:35 localhost-live kernel: BTRFS info (device sda3): ignoring data csums Feb 17 11:24:35 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots Feb 17 11:24:35 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time Feb 17 11:24:35 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled Feb 17 11:24:35 localhost-live kernel: BTRFS info (device sda3): has skinny extents Feb 17 11:24:35 localhost-live kernel: BUG: kernel NULL pointer dereference, address: 0000000000000030 Feb 17 11:24:35 localhost-live kernel: #PF: supervisor read access in kernel mode Feb 17 11:24:35 localhost-live kernel: #PF: error_code(0x0000) - not-present page Feb 17 11:24:35 localhost-live kernel: PGD 0 P4D 0 Feb 17 11:24:35 localhost-live kernel: Oops: 0000 [#1] SMP PTI Feb 17 11:24:35 localhost-live kernel: CPU: 0 PID: 4095 Comm: mount Not tainted 5.11.0-0.rc7.149.fc34.x86_64 #1 Feb 17 11:24:35 localhost-live kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020 Feb 17 11:24:35 localhost-live kernel: RIP: 0010:btrfs_device_init_dev_stats+0x4c/0x1f0 Feb 17 11:24:35 localhost-live kernel: Code: 01 00 00 53 48 83 ec 40 48 8b 47 78 48 8d 54 24 2f c6 44 24 37 f9 48 89 44 24 38 48 8b 47 38 31 ff 48 c7 44 24 2f 00 00 00 00 <48> 8b 70 30 e8 0b b4 fb ff 41 89 c5 85 c0 74 52 49 8d 84 24 40 01 Feb 17 11:24:35 localhost-live kernel: RSP: 0018:ffffa60285fbfb68 EFLAGS: 00010246 Feb 17 11:24:35 localhost-live kernel: RAX: 0000000000000000 RBX: ffff88b88f806498 RCX: ffff88b82e7a2a10 Feb 17 11:24:35 localhost-live kernel: RDX: ffffa60285fbfb97 RSI: ffff88b82e7a2a10 RDI: 0000000000000000 Feb 17 11:24:35 localhost-live kernel: RBP: ffff88b88f806b3c R08: 0000000000000000 R09: 0000000000000000 Feb 17 11:24:35 localhost-live kernel: R10: ffff88b82e7a2a10 R11: 0000000000000000 R12: ffff88b88f806a00 Feb 17 11:24:35 localhost-live kernel: R13: ffff88b88f806478 R14: ffff88b88f806a00 R15: ffff88b82e7a2a10 Feb 17 11:24:35 localhost-live kernel: FS: 00007f698be1ec40(0000) GS:ffff88b937e00000(0000) knlGS:0000000000000000 Feb 17 11:24:35 localhost-live kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Feb 17 11:24:35 localhost-live kernel: CR2: 0000000000000030 CR3: 0000000092c9c006 CR4: 00000000003706f0 Feb 17 11:24:35 localhost-live kernel: Call Trace: Feb 17 11:24:35 localhost-live kernel: ? btrfs_init_dev_stats+0x1f/0xf0 Feb 17 11:24:35 localhost-live kernel: btrfs_init_dev_stats+0x62/0xf0 Feb 17 11:24:35 localhost-live kernel: open_ctree+0x1019/0x15ff Feb 17 11:24:35 localhost-live kernel: btrfs_mount_root.cold+0x13/0xfa Feb 17 11:24:35 localhost-live kernel: legacy_get_tree+0x27/0x40 Feb 17 11:24:35 localhost-live kernel: vfs_get_tree+0x25/0xb0 Feb 17 11:24:35 localhost-live kernel: vfs_kern_mount.part.0+0x71/0xb0 Feb 17 11:24:35 localhost-live kernel: btrfs_mount+0x131/0x3d0 Feb 17 11:24:35 localhost-live kernel: ? legacy_get_tree+0x27/0x40 Feb 17 11:24:35 localhost-live kernel: ? btrfs_show_options+0x640/0x640 Feb 17 11:24:35 localhost-live kernel: legacy_get_tree+0x27/0x40 Feb 17 11:24:35 localhost-live kernel: vfs_get_tree+0x25/0xb0 Feb 17 11:24:35 localhost-live kernel: path_mount+0x441/0xa80 Feb 17 11:24:35 localhost-live kernel: __x64_sys_mount+0xf4/0x130 Feb 17 11:24:35 localhost-live kernel: do_syscall_64+0x33/0x40 Feb 17 11:24:35 localhost-live kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Feb 17 11:24:35 localhost-live kernel: RIP: 0033:0x7f698c04e52e Feb 17 11:24:35 localhost-live kernel: Code: 48 8b 0d 45 19 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 12 19 0c 00 f7 d8 64 89 01 48 Feb 17 11:24:35 localhost-live kernel: RSP: 002b:00007ffdf52bc518 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5 Feb 17 11:24:35 localhost-live kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f698c04e52e Feb 17 11:24:35 localhost-live kernel: RDX: 00005603f068e690 RSI: 00005603f068e730 RDI: 00005603f068e6b0 Feb 17 11:24:35 localhost-live kernel: RBP: 00005603f068e460 R08: 00005603f068e6f0 R09: 00007f698c110a60 Feb 17 11:24:35 localhost-live kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000 Feb 17 11:24:35 localhost-live kernel: R13: 00005603f068e6b0 R14: 00005603f068e690 R15: 00005603f068e460 Feb 17 11:24:35 localhost-live kernel: Modules linked in: snd_seq_dummy snd_hrtimer uinput nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ip_set nf_tables rfkill nfnetlink ip6table_filter ip6_tables iptable_filter vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_ac97_codec ac97_bus snd_rawmidi snd_seq snd_seq_device snd_pcm intel_rapl_msr snd_timer intel_rapl_common snd pktcdvd vmwgfx rapl soundcore gameport ttm drm_kms_helper vmw_balloon cec pcspkr joydev vmw_vmci i2c_piix4 drm zram ip_tables nls_utf8 isofs squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel serio_raw mptspi mptscsih vmxnet3 mptbase scsi_transport_spi ata_generic pata_acpi sunrpc be2iscsi bnx2i cnic uio Feb 17 11:24:35 localhost-live kernel: cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi loop fuse scsi_transport_iscsi Feb 17 11:24:35 localhost-live kernel: CR2: 0000000000000030 Feb 17 11:24:35 localhost-live kernel: ---[ end trace a8d9dde9a9afac6b ]--- Feb 17 11:24:35 localhost-live kernel: RIP: 0010:btrfs_device_init_dev_stats+0x4c/0x1f0 Feb 17 11:24:35 localhost-live kernel: Code: 01 00 00 53 48 83 ec 40 48 8b 47 78 48 8d 54 24 2f c6 44 24 37 f9 48 89 44 24 38 48 8b 47 38 31 ff 48 c7 44 24 2f 00 00 00 00 <48> 8b 70 30 e8 0b b4 fb ff 41 89 c5 85 c0 74 52 49 8d 84 24 40 01 Feb 17 11:24:35 localhost-live kernel: RSP: 0018:ffffa60285fbfb68 EFLAGS: 00010246 Feb 17 11:24:35 localhost-live kernel: RAX: 0000000000000000 RBX: ffff88b88f806498 RCX: ffff88b82e7a2a10 Feb 17 11:24:35 localhost-live kernel: RDX: ffffa60285fbfb97 RSI: ffff88b82e7a2a10 RDI: 0000000000000000 Feb 17 11:24:35 localhost-live kernel: RBP: ffff88b88f806b3c R08: 0000000000000000 R09: 0000000000000000 Feb 17 11:24:35 localhost-live kernel: R10: ffff88b82e7a2a10 R11: 0000000000000000 R12: ffff88b88f806a00 Feb 17 11:24:35 localhost-live kernel: R13: ffff88b88f806478 R14: ffff88b88f806a00 R15: ffff88b82e7a2a10 Feb 17 11:24:35 localhost-live kernel: FS: 00007f698be1ec40(0000) GS:ffff88b937e00000(0000) knlGS:0000000000000000 Feb 17 11:24:35 localhost-live kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Feb 17 11:24:35 localhost-live kernel: CR2: 0000000000000030 CR3: 0000000092c9c006 CR4: 00000000003706f0 ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-17 16:29 ` Neal Gompa @ 2021-02-17 16:44 ` Josef Bacik 2021-02-21 18:27 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-02-17 16:44 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 2/17/21 11:29 AM, Neal Gompa wrote: > On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 2/17/21 9:50 AM, Neal Gompa wrote: >>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>> >>>> On 2/16/21 9:05 PM, Neal Gompa wrote: >>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>> >>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: >>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>> >>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: >>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>> >>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: >>>>>>>>>>> Hey all, >>>>>>>>>>> >>>>>>>>>>> So one of my main computers recently had a disk controller failure >>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the >>>>>>>>>>> journal: >>>>>>>>>>> >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>> >>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>>>>>>>>>> seem to find any reasonably good information on how to do recovery in >>>>>>>>>>> this scenario, even to just recover enough to copy data off. >>>>>>>>>>> >>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>>>>>>>>>> using btrfs-progs v5.10. >>>>>>>>>>> >>>>>>>>>>> Can anyone help? >>>>>>>>>> >>>>>>>>>> Can you try >>>>>>>>>> >>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever >>>>>>>>>> >>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree >>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some >>>>>>>>>> other magic. Thanks, >>>>>>>>>> >>>>>>>>>> Josef >>>>>>>>> >>>>>>>>> I got the same error as I did with btrfs-check --readonly... >>>>>>>>> >>>>>>>> >>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? >>>>>>>> >>>>>>> >>>>>>> No dice... >>>>>>> >>>>>>> # btrfs check --readonly --backup /dev/sda3 >>>>>>>> Opening filesystem to check... >>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>> >>>>>> Hey look the block we're looking for, I wrote you some magic, just pull >>>>>> >>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal >>>>>> >>>>>> build, and then run >>>>>> >>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 >>>>>> >>>>>> This will force us to point at the old root with (hopefully) the right bytenr >>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind >>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, >>>>>> >>>>> >>>>> # btrfs check --readonly /dev/sda3 >>>>>> Opening filesystem to check... >>>>>> ERROR: could not setup extent tree >>>>>> ERROR: cannot open file system >>>>> # btrfs check --clear-space-cache v1 /dev/sda3 >>>>>> Opening filesystem to check... >>>>>> ERROR: could not setup extent tree >>>>>> ERROR: cannot open file system >>>>> >>>>> It's better, but still no dice... :( >>>>> >>>>> >>>> >>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. >>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, >>>> >>> >>> Nope, I see this in the journal: >>> >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>> >>> >> >> Ok git pull for-neal, rebuild, then run >> >> btrfs-neal-magic /dev/sda3 791281664 888895 2 >> >> I thought of this yesterday but in my head was like "naaahhhh, whats the chances >> that the level doesn't match??". Thanks, >> > > Tried rescue mount again after running that and got a stack trace in > the kernel, detailed in the following attached log. Huh I wonder how I didn't hit this when testing, I must have only tested with zero'ing the extent root and the csum root. You're going to have to build a kernel with a fix for this https://paste.centos.org/view/7b48aaea and see if that gets you further. Thanks, Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-17 16:44 ` Josef Bacik @ 2021-02-21 18:27 ` Neal Gompa 2021-02-22 19:34 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-02-21 18:27 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS [-- Attachment #1: Type: text/plain, Size: 7119 bytes --] On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > > On 2/17/21 11:29 AM, Neal Gompa wrote: > > On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 2/17/21 9:50 AM, Neal Gompa wrote: > >>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>> > >>>> On 2/16/21 9:05 PM, Neal Gompa wrote: > >>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>> > >>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: > >>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>> > >>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>> > >>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>>>>>>>> Hey all, > >>>>>>>>>>> > >>>>>>>>>>> So one of my main computers recently had a disk controller failure > >>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the > >>>>>>>>>>> journal: > >>>>>>>>>>> > >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>> > >>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>>>>>>>>>> seem to find any reasonably good information on how to do recovery in > >>>>>>>>>>> this scenario, even to just recover enough to copy data off. > >>>>>>>>>>> > >>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>>>>>>>>>> using btrfs-progs v5.10. > >>>>>>>>>>> > >>>>>>>>>>> Can anyone help? > >>>>>>>>>> > >>>>>>>>>> Can you try > >>>>>>>>>> > >>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>>>>>>>> > >>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree > >>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some > >>>>>>>>>> other magic. Thanks, > >>>>>>>>>> > >>>>>>>>>> Josef > >>>>>>>>> > >>>>>>>>> I got the same error as I did with btrfs-check --readonly... > >>>>>>>>> > >>>>>>>> > >>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? > >>>>>>>> > >>>>>>> > >>>>>>> No dice... > >>>>>>> > >>>>>>> # btrfs check --readonly --backup /dev/sda3 > >>>>>>>> Opening filesystem to check... > >>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>> > >>>>>> Hey look the block we're looking for, I wrote you some magic, just pull > >>>>>> > >>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal > >>>>>> > >>>>>> build, and then run > >>>>>> > >>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 > >>>>>> > >>>>>> This will force us to point at the old root with (hopefully) the right bytenr > >>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind > >>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, > >>>>>> > >>>>> > >>>>> # btrfs check --readonly /dev/sda3 > >>>>>> Opening filesystem to check... > >>>>>> ERROR: could not setup extent tree > >>>>>> ERROR: cannot open file system > >>>>> # btrfs check --clear-space-cache v1 /dev/sda3 > >>>>>> Opening filesystem to check... > >>>>>> ERROR: could not setup extent tree > >>>>>> ERROR: cannot open file system > >>>>> > >>>>> It's better, but still no dice... :( > >>>>> > >>>>> > >>>> > >>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. > >>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, > >>>> > >>> > >>> Nope, I see this in the journal: > >>> > >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options > >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums > >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots > >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time > >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>> > >>> > >> > >> Ok git pull for-neal, rebuild, then run > >> > >> btrfs-neal-magic /dev/sda3 791281664 888895 2 > >> > >> I thought of this yesterday but in my head was like "naaahhhh, whats the chances > >> that the level doesn't match??". Thanks, > >> > > > > Tried rescue mount again after running that and got a stack trace in > > the kernel, detailed in the following attached log. > > Huh I wonder how I didn't hit this when testing, I must have only tested with > zero'ing the extent root and the csum root. You're going to have to build a > kernel with a fix for this > > https://paste.centos.org/view/7b48aaea > > and see if that gets you further. Thanks, > I built a kernel build as an RPM with your patch[1] and tried it. [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt Killed The log from the journal is attached. [1]: https://download.copr.fedorainfracloud.org/results/ngompa/btrfs-progs-neal-magic/fedora-34-x86_64/01987802-kernel/ -- 真実はいつも一つ！/ Always, there's only one truth! [-- Attachment #2: output-clean.log --] [-- Type: text/x-log, Size: 6787 bytes --] Feb 21 13:18:58 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options Feb 21 13:18:58 fedora kernel: BTRFS info (device sdb3): ignoring data csums Feb 21 13:18:58 fedora kernel: BTRFS info (device sdb3): ignoring bad roots Feb 21 13:18:58 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time Feb 21 13:18:58 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled Feb 21 13:18:58 fedora kernel: BTRFS info (device sdb3): has skinny extents Feb 21 13:18:58 fedora kernel: BUG: kernel NULL pointer dereference, address: 0000000000000030 Feb 21 13:18:58 fedora kernel: #PF: supervisor read access in kernel mode Feb 21 13:18:58 fedora kernel: #PF: error_code(0x0000) - not-present page Feb 21 13:18:58 fedora kernel: PGD 0 P4D 0 Feb 21 13:18:58 fedora kernel: Oops: 0000 [#1] SMP PTI Feb 21 13:18:58 fedora kernel: CPU: 1 PID: 1590 Comm: mount Not tainted 5.11.0-155.nealbtrfstest.fc34.x86_64 #1 Feb 21 13:18:58 fedora kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020 Feb 21 13:18:58 fedora kernel: RIP: 0010:btrfs_device_init_dev_stats+0x26/0x210 Feb 21 13:18:58 fedora kernel: Code: 0f 1f 40 00 0f 1f 44 00 00 41 57 49 89 f7 41 56 41 55 45 31 ed 41 54 55 53 48 83 ec 40 48 8b 47 38 48 c7 44 24 2f 00 00 00 00 <48> 8b 70 30 c6 44 24 3f 00 48 c7 44 24 37 00 00 00 00 48 85 f6 74 Feb 21 13:18:58 fedora kernel: RSP: 0018:ffffb477c3ce3b68 EFLAGS: 00010282 Feb 21 13:18:58 fedora kernel: RAX: 0000000000000000 RBX: ffff8ad3094ea098 RCX: 0000000000000070 Feb 21 13:18:58 fedora kernel: RDX: ffff8ad31b728000 RSI: ffff8ad328c8c2a0 RDI: ffff8ad3094eac00 Feb 21 13:18:58 fedora kernel: RBP: ffff8ad328c8c2a0 R08: 0000000000000070 R09: 0000000000000000 Feb 21 13:18:58 fedora kernel: R10: ffff8ad328c8c2a0 R11: 0000000000000000 R12: ffff8ad3094ea000 Feb 21 13:18:58 fedora kernel: R13: 0000000000000000 R14: ffff8ad3094eac00 R15: ffff8ad328c8c2a0 Feb 21 13:18:58 fedora kernel: FS: 00007fda4979fc40(0000) GS:ffff8ad37be40000(0000) knlGS:0000000000000000 Feb 21 13:18:58 fedora kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Feb 21 13:18:58 fedora kernel: CR2: 0000000000000030 CR3: 000000005faca001 CR4: 00000000003706e0 Feb 21 13:18:58 fedora kernel: Call Trace: Feb 21 13:18:58 fedora kernel: ? btrfs_init_dev_stats+0x1f/0xf0 Feb 21 13:18:58 fedora kernel: btrfs_init_dev_stats+0x62/0xf0 Feb 21 13:18:58 fedora kernel: open_ctree+0x102c/0x1610 Feb 21 13:18:58 fedora kernel: btrfs_mount_root.cold+0x13/0xfa Feb 21 13:18:58 fedora kernel: legacy_get_tree+0x27/0x40 Feb 21 13:18:58 fedora kernel: vfs_get_tree+0x25/0xb0 Feb 21 13:18:58 fedora kernel: vfs_kern_mount.part.0+0x71/0xb0 Feb 21 13:18:58 fedora kernel: btrfs_mount+0x131/0x3d0 Feb 21 13:18:58 fedora kernel: ? legacy_get_tree+0x27/0x40 Feb 21 13:18:58 fedora kernel: ? btrfs_show_options+0x640/0x640 Feb 21 13:18:58 fedora kernel: legacy_get_tree+0x27/0x40 Feb 21 13:18:58 fedora kernel: vfs_get_tree+0x25/0xb0 Feb 21 13:18:58 fedora kernel: path_mount+0x441/0xa80 Feb 21 13:18:58 fedora kernel: __x64_sys_mount+0xf4/0x130 Feb 21 13:18:58 fedora kernel: do_syscall_64+0x33/0x40 Feb 21 13:18:58 fedora kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Feb 21 13:18:58 fedora kernel: RIP: 0033:0x7fda499cf52e Feb 21 13:18:58 fedora kernel: Code: 48 8b 0d 45 19 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 12 19 0c 00 f7 d8 64 89 01 48 Feb 21 13:18:58 fedora kernel: RSP: 002b:00007ffc12dff688 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5 Feb 21 13:18:58 fedora kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fda499cf52e Feb 21 13:18:58 fedora kernel: RDX: 000055e2fc6f3690 RSI: 000055e2fc6f3730 RDI: 000055e2fc6f36b0 Feb 21 13:18:58 fedora kernel: RBP: 000055e2fc6f3460 R08: 000055e2fc6f36f0 R09: 00007fda49a91a60 Feb 21 13:18:58 fedora kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000 Feb 21 13:18:58 fedora kernel: R13: 000055e2fc6f36b0 R14: 000055e2fc6f3690 R15: 000055e2fc6f3460 Feb 21 13:18:58 fedora kernel: Modules linked in: bnep snd_seq_dummy snd_hrtimer bluetooth ecdh_generic ecc nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ip_set nf_tables rfkill nfnetlink ip6table_filter ip6_tables iptable_filter vsock_loopback vmw_vsock_virtio_transport_common snd_seq_midi snd_seq_midi_event vmw_vsock_vmci_transport vsock sunrpc intel_rapl_msr intel_rapl_common rapl vmw_balloon snd_ens1371 snd_ac97_codec ac97_bus snd_rawmidi snd_seq snd_seq_device snd_pcm joydev pcspkr snd_timer snd soundcore gameport vmw_vmci i2c_piix4 zram ip_tables crct10dif_pclmul crc32_pclmul vmwgfx crc32c_intel drm_kms_helper ghash_clmulni_intel mptspi e1000 cec ttm scsi_transport_spi serio_raw drm mptscsih mptbase ata_generic pata_acpi fuse Feb 21 13:18:58 fedora kernel: CR2: 0000000000000030 Feb 21 13:18:58 fedora kernel: ---[ end trace 87ac94f887eabb67 ]--- Feb 21 13:18:58 fedora kernel: RIP: 0010:btrfs_device_init_dev_stats+0x26/0x210 Feb 21 13:18:58 fedora kernel: Code: 0f 1f 40 00 0f 1f 44 00 00 41 57 49 89 f7 41 56 41 55 45 31 ed 41 54 55 53 48 83 ec 40 48 8b 47 38 48 c7 44 24 2f 00 00 00 00 <48> 8b 70 30 c6 44 24 3f 00 48 c7 44 24 37 00 00 00 00 48 85 f6 74 Feb 21 13:18:58 fedora kernel: RSP: 0018:ffffb477c3ce3b68 EFLAGS: 00010282 Feb 21 13:18:58 fedora kernel: RAX: 0000000000000000 RBX: ffff8ad3094ea098 RCX: 0000000000000070 Feb 21 13:18:58 fedora kernel: RDX: ffff8ad31b728000 RSI: ffff8ad328c8c2a0 RDI: ffff8ad3094eac00 Feb 21 13:18:58 fedora kernel: RBP: ffff8ad328c8c2a0 R08: 0000000000000070 R09: 0000000000000000 Feb 21 13:18:58 fedora kernel: R10: ffff8ad328c8c2a0 R11: 0000000000000000 R12: ffff8ad3094ea000 Feb 21 13:18:58 fedora kernel: R13: 0000000000000000 R14: ffff8ad3094eac00 R15: ffff8ad328c8c2a0 Feb 21 13:18:58 fedora kernel: FS: 00007fda4979fc40(0000) GS:ffff8ad37be40000(0000) knlGS:0000000000000000 Feb 21 13:18:58 fedora kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Feb 21 13:18:58 fedora kernel: CR2: 0000000000000030 CR3: 000000005faca001 CR4: 00000000003706e0 Feb 21 13:18:59 fedora abrt-dump-journal-oops[700]: abrt-dump-journal-oops: Found oopses: 1 Feb 21 13:18:59 fedora abrt-dump-journal-oops[700]: abrt-dump-journal-oops: Creating problem directories Feb 21 13:19:00 fedora abrt-notification[1631]: System encountered a non-fatal error in btrfs_init_dev_stats() Feb 21 13:19:00 fedora abrt-dump-journal-oops[700]: Reported 1 kernel oopses to Abrt ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-21 18:27 ` Neal Gompa @ 2021-02-22 19:34 ` Josef Bacik 2021-02-23 4:03 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-02-22 19:34 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 2/21/21 1:27 PM, Neal Gompa wrote: > On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 2/17/21 11:29 AM, Neal Gompa wrote: >>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>> >>>> On 2/17/21 9:50 AM, Neal Gompa wrote: >>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>> >>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: >>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>> >>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: >>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>> >>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: >>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: >>>>>>>>>>>>> Hey all, >>>>>>>>>>>>> >>>>>>>>>>>>> So one of my main computers recently had a disk controller failure >>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the >>>>>>>>>>>>> journal: >>>>>>>>>>>>> >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>> >>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in >>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. >>>>>>>>>>>>> >>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>>>>>>>>>>>> using btrfs-progs v5.10. >>>>>>>>>>>>> >>>>>>>>>>>>> Can anyone help? >>>>>>>>>>>> >>>>>>>>>>>> Can you try >>>>>>>>>>>> >>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever >>>>>>>>>>>> >>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree >>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some >>>>>>>>>>>> other magic. Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Josef >>>>>>>>>>> >>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? >>>>>>>>>> >>>>>>>>> >>>>>>>>> No dice... >>>>>>>>> >>>>>>>>> # btrfs check --readonly --backup /dev/sda3 >>>>>>>>>> Opening filesystem to check... >>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>> >>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull >>>>>>>> >>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal >>>>>>>> >>>>>>>> build, and then run >>>>>>>> >>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 >>>>>>>> >>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr >>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind >>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, >>>>>>>> >>>>>>> >>>>>>> # btrfs check --readonly /dev/sda3 >>>>>>>> Opening filesystem to check... >>>>>>>> ERROR: could not setup extent tree >>>>>>>> ERROR: cannot open file system >>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 >>>>>>>> Opening filesystem to check... >>>>>>>> ERROR: could not setup extent tree >>>>>>>> ERROR: cannot open file system >>>>>>> >>>>>>> It's better, but still no dice... :( >>>>>>> >>>>>>> >>>>>> >>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. >>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, >>>>>> >>>>> >>>>> Nope, I see this in the journal: >>>>> >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>> >>>>> >>>> >>>> Ok git pull for-neal, rebuild, then run >>>> >>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 >>>> >>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances >>>> that the level doesn't match??". Thanks, >>>> >>> >>> Tried rescue mount again after running that and got a stack trace in >>> the kernel, detailed in the following attached log. >> >> Huh I wonder how I didn't hit this when testing, I must have only tested with >> zero'ing the extent root and the csum root. You're going to have to build a >> kernel with a fix for this >> >> https://paste.centos.org/view/7b48aaea >> >> and see if that gets you further. Thanks, >> > > I built a kernel build as an RPM with your patch[1] and tried it. > > [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > Killed > > The log from the journal is attached. Ahh crud my bad, this should do it https://paste.centos.org/view/ac2e61ef Thanks, Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-22 19:34 ` Josef Bacik @ 2021-02-23 4:03 ` Neal Gompa 2021-02-23 15:05 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-02-23 4:03 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: > > On 2/21/21 1:27 PM, Neal Gompa wrote: > > On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 2/17/21 11:29 AM, Neal Gompa wrote: > >>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>> > >>>> On 2/17/21 9:50 AM, Neal Gompa wrote: > >>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>> > >>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: > >>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>> > >>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: > >>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>> > >>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>>>>>>>>>> Hey all, > >>>>>>>>>>>>> > >>>>>>>>>>>>> So one of my main computers recently had a disk controller failure > >>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the > >>>>>>>>>>>>> journal: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>> > >>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in > >>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>>>>>>>>>>>> using btrfs-progs v5.10. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Can anyone help? > >>>>>>>>>>>> > >>>>>>>>>>>> Can you try > >>>>>>>>>>>> > >>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>>>>>>>>>> > >>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree > >>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some > >>>>>>>>>>>> other magic. Thanks, > >>>>>>>>>>>> > >>>>>>>>>>>> Josef > >>>>>>>>>>> > >>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> No dice... > >>>>>>>>> > >>>>>>>>> # btrfs check --readonly --backup /dev/sda3 > >>>>>>>>>> Opening filesystem to check... > >>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>> > >>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull > >>>>>>>> > >>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal > >>>>>>>> > >>>>>>>> build, and then run > >>>>>>>> > >>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 > >>>>>>>> > >>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr > >>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind > >>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, > >>>>>>>> > >>>>>>> > >>>>>>> # btrfs check --readonly /dev/sda3 > >>>>>>>> Opening filesystem to check... > >>>>>>>> ERROR: could not setup extent tree > >>>>>>>> ERROR: cannot open file system > >>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 > >>>>>>>> Opening filesystem to check... > >>>>>>>> ERROR: could not setup extent tree > >>>>>>>> ERROR: cannot open file system > >>>>>>> > >>>>>>> It's better, but still no dice... :( > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. > >>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, > >>>>>> > >>>>> > >>>>> Nope, I see this in the journal: > >>>>> > >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options > >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums > >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots > >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time > >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>> > >>>>> > >>>> > >>>> Ok git pull for-neal, rebuild, then run > >>>> > >>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 > >>>> > >>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances > >>>> that the level doesn't match??". Thanks, > >>>> > >>> > >>> Tried rescue mount again after running that and got a stack trace in > >>> the kernel, detailed in the following attached log. > >> > >> Huh I wonder how I didn't hit this when testing, I must have only tested with > >> zero'ing the extent root and the csum root. You're going to have to build a > >> kernel with a fix for this > >> > >> https://paste.centos.org/view/7b48aaea > >> > >> and see if that gets you further. Thanks, > >> > > > > I built a kernel build as an RPM with your patch[1] and tried it. > > > > [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > > Killed > > > > The log from the journal is attached. > > > Ahh crud my bad, this should do it > > https://paste.centos.org/view/ac2e61ef > Patch doesn't apply (note it is patch 667 below): ngompa@fedora-rawhide-skuldvm ~/f/kernel (rawhide)> fedpkg prep setting SOURCE_DATE_EPOCH=1613347200 Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.f5LOt6 + umask 022 + cd /home/ngompa/fedora-scm/kernel + patch_command='patch -p1 -F1 -s' + cd /home/ngompa/fedora-scm/kernel + rm -rf kernel-5.11 + /usr/bin/mkdir -p kernel-5.11 + cd kernel-5.11 + /usr/bin/xz -dc /home/ngompa/fedora-scm/kernel/linux-5.11.tar.xz + /usr/bin/tar -xof - + STATUS=0 + '[' 0 -ne 0 ']' + /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w . + mv linux-5.11 linux-5.11.0-155.nealbtrfstest.1.fc35.x86_64 + cd linux-5.11.0-155.nealbtrfstest.1.fc35.x86_64 + cp -a /home/ngompa/fedora-scm/kernel/Makefile.rhelver . + ApplyOptionalPatch patch-5.11.0-redhat.patch + local patch=patch-5.11.0-redhat.patch + shift + '[' '!' -f /home/ngompa/fedora-scm/kernel/patch-5.11.0-redhat.patch ']' ++ awk '{print $1}' ++ wc -l /home/ngompa/fedora-scm/kernel/patch-5.11.0-redhat.patch + local C=3166 + '[' 3166 -gt 9 ']' + ApplyPatch patch-5.11.0-redhat.patch + local patch=patch-5.11.0-redhat.patch + shift + '[' '!' -f /home/ngompa/fedora-scm/kernel/patch-5.11.0-redhat.patch ']' + case "$patch" in + patch -p1 -F1 -s + echo 'Patch #666 (linux-5.11-btrfs-handle-null-roots.diff):' Patch #666 (linux-5.11-btrfs-handle-null-roots.diff): + /usr/bin/patch --no-backup-if-mismatch -p1 --fuzz=0 patching file fs/btrfs/ctree.c Hunk #1 succeeded at 2594 (offset -4 lines). patching file fs/btrfs/volumes.c Hunk #1 succeeded at 7282 (offset -166 lines). + echo 'Patch #667 (linux-5.11-btrfs-init-devices-more-gracefully.diff):' Patch #667 (linux-5.11-btrfs-init-devices-more-gracefully.diff): + /usr/bin/patch --no-backup-if-mismatch -p1 --fuzz=0 patching file fs/btrfs/disk-io.c patching file fs/btrfs/volumes.c Hunk #1 FAILED at 7282. 1 out of 1 hunk FAILED -- saving rejects to file fs/btrfs/volumes.c.rej error: Bad exit status from /var/tmp/rpm-tmp.f5LOt6 (%prep) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.f5LOt6 (%prep) Could not execute prep: Failed to execute command. -- 真実はいつも一つ！/ Always, there's only one truth! ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-23 4:03 ` Neal Gompa @ 2021-02-23 15:05 ` Josef Bacik 2021-02-24 14:23 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-02-23 15:05 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 2/22/21 11:03 PM, Neal Gompa wrote: > On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 2/21/21 1:27 PM, Neal Gompa wrote: >>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>> >>>> On 2/17/21 11:29 AM, Neal Gompa wrote: >>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>> >>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: >>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>> >>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: >>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>> >>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: >>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: >>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: >>>>>>>>>>>>>>> Hey all, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure >>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the >>>>>>>>>>>>>>> journal: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in >>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>>>>>>>>>>>>>> using btrfs-progs v5.10. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can anyone help? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can you try >>>>>>>>>>>>>> >>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever >>>>>>>>>>>>>> >>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree >>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some >>>>>>>>>>>>>> other magic. Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Josef >>>>>>>>>>>>> >>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> No dice... >>>>>>>>>>> >>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 >>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>> >>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull >>>>>>>>>> >>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal >>>>>>>>>> >>>>>>>>>> build, and then run >>>>>>>>>> >>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 >>>>>>>>>> >>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr >>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind >>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, >>>>>>>>>> >>>>>>>>> >>>>>>>>> # btrfs check --readonly /dev/sda3 >>>>>>>>>> Opening filesystem to check... >>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>> ERROR: cannot open file system >>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 >>>>>>>>>> Opening filesystem to check... >>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>> ERROR: cannot open file system >>>>>>>>> >>>>>>>>> It's better, but still no dice... :( >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. >>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, >>>>>>>> >>>>>>> >>>>>>> Nope, I see this in the journal: >>>>>>> >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>> >>>>>>> >>>>>> >>>>>> Ok git pull for-neal, rebuild, then run >>>>>> >>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 >>>>>> >>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances >>>>>> that the level doesn't match??". Thanks, >>>>>> >>>>> >>>>> Tried rescue mount again after running that and got a stack trace in >>>>> the kernel, detailed in the following attached log. >>>> >>>> Huh I wonder how I didn't hit this when testing, I must have only tested with >>>> zero'ing the extent root and the csum root. You're going to have to build a >>>> kernel with a fix for this >>>> >>>> https://paste.centos.org/view/7b48aaea >>>> >>>> and see if that gets you further. Thanks, >>>> >>> >>> I built a kernel build as an RPM with your patch[1] and tried it. >>> >>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>> Killed >>> >>> The log from the journal is attached. >> >> >> Ahh crud my bad, this should do it >> >> https://paste.centos.org/view/ac2e61ef >> > > Patch doesn't apply (note it is patch 667 below): Ah sorry, should have just sent you an iterative patch. You can take the above patch and just delete the hunk from volumes.c as you already have that applied and then it'll work. Thanks, Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-23 15:05 ` Josef Bacik @ 2021-02-24 14:23 ` Neal Gompa 2021-02-24 15:44 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-02-24 14:23 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS [-- Attachment #1: Type: text/plain, Size: 8412 bytes --] On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: > > On 2/22/21 11:03 PM, Neal Gompa wrote: > > On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 2/21/21 1:27 PM, Neal Gompa wrote: > >>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>> > >>>> On 2/17/21 11:29 AM, Neal Gompa wrote: > >>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>> > >>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: > >>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>> > >>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: > >>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>> > >>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: > >>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>> Hey all, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure > >>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the > >>>>>>>>>>>>>>> journal: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in > >>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>>>>>>>>>>>>>> using btrfs-progs v5.10. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Can anyone help? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Can you try > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree > >>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some > >>>>>>>>>>>>>> other magic. Thanks, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Josef > >>>>>>>>>>>>> > >>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> No dice... > >>>>>>>>>>> > >>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 > >>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>> > >>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull > >>>>>>>>>> > >>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal > >>>>>>>>>> > >>>>>>>>>> build, and then run > >>>>>>>>>> > >>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 > >>>>>>>>>> > >>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr > >>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind > >>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> # btrfs check --readonly /dev/sda3 > >>>>>>>>>> Opening filesystem to check... > >>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>> ERROR: cannot open file system > >>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 > >>>>>>>>>> Opening filesystem to check... > >>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>> ERROR: cannot open file system > >>>>>>>>> > >>>>>>>>> It's better, but still no dice... :( > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. > >>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, > >>>>>>>> > >>>>>>> > >>>>>>> Nope, I see this in the journal: > >>>>>>> > >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options > >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums > >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots > >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time > >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> Ok git pull for-neal, rebuild, then run > >>>>>> > >>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 > >>>>>> > >>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances > >>>>>> that the level doesn't match??". Thanks, > >>>>>> > >>>>> > >>>>> Tried rescue mount again after running that and got a stack trace in > >>>>> the kernel, detailed in the following attached log. > >>>> > >>>> Huh I wonder how I didn't hit this when testing, I must have only tested with > >>>> zero'ing the extent root and the csum root. You're going to have to build a > >>>> kernel with a fix for this > >>>> > >>>> https://paste.centos.org/view/7b48aaea > >>>> > >>>> and see if that gets you further. Thanks, > >>>> > >>> > >>> I built a kernel build as an RPM with your patch[1] and tried it. > >>> > >>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>> Killed > >>> > >>> The log from the journal is attached. > >> > >> > >> Ahh crud my bad, this should do it > >> > >> https://paste.centos.org/view/ac2e61ef > >> > > > > Patch doesn't apply (note it is patch 667 below): > > Ah sorry, should have just sent you an iterative patch. You can take the above > patch and just delete the hunk from volumes.c as you already have that applied > and then it'll work. Thanks, > Failed with a weird error...? [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt mount: /mnt: mount(2) system call failed: No such file or directory. Journal log with traceback attached. -- 真実はいつも一つ！/ Always, there's only one truth! [-- Attachment #2: output-clean2.log --] [-- Type: text/x-log, Size: 2987 bytes --] Feb 24 09:12:41 fedora kernel: BTRFS info (device sda3): enabling all of the rescue options Feb 24 09:12:41 fedora kernel: BTRFS info (device sda3): ignoring data csums Feb 24 09:12:41 fedora kernel: BTRFS info (device sda3): ignoring bad roots Feb 24 09:12:41 fedora kernel: BTRFS info (device sda3): disabling log replay at mount time Feb 24 09:12:41 fedora kernel: BTRFS info (device sda3): disk space caching is enabled Feb 24 09:12:41 fedora kernel: BTRFS info (device sda3): has skinny extents Feb 24 09:12:41 fedora kernel: we tried to search with a NULL root Feb 24 09:12:41 fedora kernel: CPU: 0 PID: 1760 Comm: mount Not tainted 5.11.0-155.nealbtrfstest.1.fc34.x86_64 #1 Feb 24 09:12:41 fedora kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020 Feb 24 09:12:41 fedora kernel: Call Trace: Feb 24 09:12:41 fedora kernel: dump_stack+0x6b/0x83 Feb 24 09:12:41 fedora kernel: btrfs_search_slot.cold+0x11/0x1b Feb 24 09:12:41 fedora kernel: ? btrfs_init_dev_replace+0x36/0x450 Feb 24 09:12:41 fedora kernel: btrfs_init_dev_replace+0x71/0x450 Feb 24 09:12:41 fedora kernel: open_ctree+0x1054/0x1610 Feb 24 09:12:41 fedora kernel: btrfs_mount_root.cold+0x13/0xfa Feb 24 09:12:41 fedora kernel: legacy_get_tree+0x27/0x40 Feb 24 09:12:41 fedora kernel: vfs_get_tree+0x25/0xb0 Feb 24 09:12:41 fedora kernel: vfs_kern_mount.part.0+0x71/0xb0 Feb 24 09:12:41 fedora kernel: btrfs_mount+0x131/0x3d0 Feb 24 09:12:41 fedora kernel: ? legacy_get_tree+0x27/0x40 Feb 24 09:12:41 fedora kernel: ? btrfs_show_options+0x640/0x640 Feb 24 09:12:41 fedora kernel: legacy_get_tree+0x27/0x40 Feb 24 09:12:41 fedora kernel: vfs_get_tree+0x25/0xb0 Feb 24 09:12:41 fedora kernel: path_mount+0x441/0xa80 Feb 24 09:12:41 fedora kernel: __x64_sys_mount+0xf4/0x130 Feb 24 09:12:41 fedora kernel: do_syscall_64+0x33/0x40 Feb 24 09:12:41 fedora kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Feb 24 09:12:41 fedora kernel: RIP: 0033:0x7f644730352e Feb 24 09:12:41 fedora kernel: Code: 48 8b 0d 45 19 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 12 19 0c 00 f7 d8 64 89 01 48 Feb 24 09:12:41 fedora kernel: RSP: 002b:00007ffd338bfac8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5 Feb 24 09:12:41 fedora kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f644730352e Feb 24 09:12:41 fedora kernel: RDX: 000055d4c6983690 RSI: 000055d4c6983730 RDI: 000055d4c69836b0 Feb 24 09:12:41 fedora kernel: RBP: 000055d4c6983460 R08: 000055d4c69836f0 R09: 00007f64473c5a60 Feb 24 09:12:41 fedora kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000 Feb 24 09:12:41 fedora kernel: R13: 000055d4c69836b0 R14: 000055d4c6983690 R15: 000055d4c6983460 Feb 24 09:12:41 fedora kernel: BTRFS warning (device sda3): failed to read fs tree: -2 Feb 24 09:12:41 fedora kernel: BTRFS error (device sda3): open_ctree failed ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-24 14:23 ` Neal Gompa @ 2021-02-24 15:44 ` Josef Bacik 2021-02-25 3:47 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-02-24 15:44 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 2/24/21 9:23 AM, Neal Gompa wrote: > On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 2/22/21 11:03 PM, Neal Gompa wrote: >>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>> >>>> On 2/21/21 1:27 PM, Neal Gompa wrote: >>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>> >>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: >>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>> >>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: >>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>> >>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: >>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: >>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: >>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>> Hey all, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure >>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the >>>>>>>>>>>>>>>>> journal: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in >>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>>>>>>>>>>>>>>>> using btrfs-progs v5.10. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Can anyone help? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Can you try >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree >>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some >>>>>>>>>>>>>>>> other magic. Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Josef >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> No dice... >>>>>>>>>>>>> >>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 >>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>> >>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull >>>>>>>>>>>> >>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal >>>>>>>>>>>> >>>>>>>>>>>> build, and then run >>>>>>>>>>>> >>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 >>>>>>>>>>>> >>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr >>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind >>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> # btrfs check --readonly /dev/sda3 >>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 >>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>> >>>>>>>>>>> It's better, but still no dice... :( >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. >>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, >>>>>>>>>> >>>>>>>>> >>>>>>>>> Nope, I see this in the journal: >>>>>>>>> >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> Ok git pull for-neal, rebuild, then run >>>>>>>> >>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 >>>>>>>> >>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances >>>>>>>> that the level doesn't match??". Thanks, >>>>>>>> >>>>>>> >>>>>>> Tried rescue mount again after running that and got a stack trace in >>>>>>> the kernel, detailed in the following attached log. >>>>>> >>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with >>>>>> zero'ing the extent root and the csum root. You're going to have to build a >>>>>> kernel with a fix for this >>>>>> >>>>>> https://paste.centos.org/view/7b48aaea >>>>>> >>>>>> and see if that gets you further. Thanks, >>>>>> >>>>> >>>>> I built a kernel build as an RPM with your patch[1] and tried it. >>>>> >>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>>>> Killed >>>>> >>>>> The log from the journal is attached. >>>> >>>> >>>> Ahh crud my bad, this should do it >>>> >>>> https://paste.centos.org/view/ac2e61ef >>>> >>> >>> Patch doesn't apply (note it is patch 667 below): >> >> Ah sorry, should have just sent you an iterative patch. You can take the above >> patch and just delete the hunk from volumes.c as you already have that applied >> and then it'll work. Thanks, >> > > Failed with a weird error...? > > [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt > mount: /mnt: mount(2) system call failed: No such file or directory. > > Journal log with traceback attached. Last one maybe? https://paste.centos.org/view/80edd6fd thanks, Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-24 15:44 ` Josef Bacik @ 2021-02-25 3:47 ` Neal Gompa 2021-03-03 18:42 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-02-25 3:47 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > > On 2/24/21 9:23 AM, Neal Gompa wrote: > > On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 2/22/21 11:03 PM, Neal Gompa wrote: > >>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>> > >>>> On 2/21/21 1:27 PM, Neal Gompa wrote: > >>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>> > >>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: > >>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>> > >>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: > >>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>> > >>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: > >>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: > >>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>> Hey all, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure > >>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the > >>>>>>>>>>>>>>>>> journal: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in > >>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>>>>>>>>>>>>>>>> using btrfs-progs v5.10. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Can anyone help? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Can you try > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree > >>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some > >>>>>>>>>>>>>>>> other magic. Thanks, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Josef > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> No dice... > >>>>>>>>>>>>> > >>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 > >>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>> > >>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull > >>>>>>>>>>>> > >>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal > >>>>>>>>>>>> > >>>>>>>>>>>> build, and then run > >>>>>>>>>>>> > >>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 > >>>>>>>>>>>> > >>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr > >>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind > >>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> # btrfs check --readonly /dev/sda3 > >>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 > >>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>> > >>>>>>>>>>> It's better, but still no dice... :( > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. > >>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> Nope, I see this in the journal: > >>>>>>>>> > >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options > >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums > >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots > >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time > >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> Ok git pull for-neal, rebuild, then run > >>>>>>>> > >>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 > >>>>>>>> > >>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances > >>>>>>>> that the level doesn't match??". Thanks, > >>>>>>>> > >>>>>>> > >>>>>>> Tried rescue mount again after running that and got a stack trace in > >>>>>>> the kernel, detailed in the following attached log. > >>>>>> > >>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with > >>>>>> zero'ing the extent root and the csum root. You're going to have to build a > >>>>>> kernel with a fix for this > >>>>>> > >>>>>> https://paste.centos.org/view/7b48aaea > >>>>>> > >>>>>> and see if that gets you further. Thanks, > >>>>>> > >>>>> > >>>>> I built a kernel build as an RPM with your patch[1] and tried it. > >>>>> > >>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>> Killed > >>>>> > >>>>> The log from the journal is attached. > >>>> > >>>> > >>>> Ahh crud my bad, this should do it > >>>> > >>>> https://paste.centos.org/view/ac2e61ef > >>>> > >>> > >>> Patch doesn't apply (note it is patch 667 below): > >> > >> Ah sorry, should have just sent you an iterative patch. You can take the above > >> patch and just delete the hunk from volumes.c as you already have that applied > >> and then it'll work. Thanks, > >> > > > > Failed with a weird error...? > > > > [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt > > mount: /mnt: mount(2) system call failed: No such file or directory. > > > > Journal log with traceback attached. > > Last one maybe? > > https://paste.centos.org/view/80edd6fd > Similar weird failure: [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt mount: /mnt: mount(2) system call failed: No such file or directory. No crash in the journal this time, though: > Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options > Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums > Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots > Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time > Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled > Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents > Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 > Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed -- 真実はいつも一つ！/ Always, there's only one truth! ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-02-25 3:47 ` Neal Gompa @ 2021-03-03 18:42 ` Josef Bacik 2021-03-03 19:38 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-03-03 18:42 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 2/24/21 10:47 PM, Neal Gompa wrote: > On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 2/24/21 9:23 AM, Neal Gompa wrote: >>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>> >>>> On 2/22/21 11:03 PM, Neal Gompa wrote: >>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>> >>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: >>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>> >>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: >>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>> >>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: >>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: >>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: >>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>> Hey all, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure >>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the >>>>>>>>>>>>>>>>>>> journal: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in >>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Can anyone help? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Can you try >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree >>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some >>>>>>>>>>>>>>>>>> other magic. Thanks, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Josef >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> No dice... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 >>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal >>>>>>>>>>>>>> >>>>>>>>>>>>>> build, and then run >>>>>>>>>>>>>> >>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 >>>>>>>>>>>>>> >>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr >>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind >>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 >>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 >>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>> >>>>>>>>>>>>> It's better, but still no dice... :( >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. >>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Nope, I see this in the journal: >>>>>>>>>>> >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Ok git pull for-neal, rebuild, then run >>>>>>>>>> >>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 >>>>>>>>>> >>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances >>>>>>>>>> that the level doesn't match??". Thanks, >>>>>>>>>> >>>>>>>>> >>>>>>>>> Tried rescue mount again after running that and got a stack trace in >>>>>>>>> the kernel, detailed in the following attached log. >>>>>>>> >>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with >>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a >>>>>>>> kernel with a fix for this >>>>>>>> >>>>>>>> https://paste.centos.org/view/7b48aaea >>>>>>>> >>>>>>>> and see if that gets you further. Thanks, >>>>>>>> >>>>>>> >>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. >>>>>>> >>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>>>>>> Killed >>>>>>> >>>>>>> The log from the journal is attached. >>>>>> >>>>>> >>>>>> Ahh crud my bad, this should do it >>>>>> >>>>>> https://paste.centos.org/view/ac2e61ef >>>>>> >>>>> >>>>> Patch doesn't apply (note it is patch 667 below): >>>> >>>> Ah sorry, should have just sent you an iterative patch. You can take the above >>>> patch and just delete the hunk from volumes.c as you already have that applied >>>> and then it'll work. Thanks, >>>> >>> >>> Failed with a weird error...? >>> >>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt >>> mount: /mnt: mount(2) system call failed: No such file or directory. >>> >>> Journal log with traceback attached. >> >> Last one maybe? >> >> https://paste.centos.org/view/80edd6fd >> > > Similar weird failure: > > [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > mount: /mnt: mount(2) system call failed: No such file or directory. > > No crash in the journal this time, though: > >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents >> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 >> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed > > Sorry Neal, you replied when I was in the middle of something and promptly forgot about it. I figured the fs root was fine, can you do the following so I can figure out from the error messages what might be wrong btrfs check --readonly btrfs restore -D btrfs restore -l Thanks, Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-03-03 18:42 ` Josef Bacik @ 2021-03-03 19:38 ` Neal Gompa 2021-03-04 20:25 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-03-03 19:38 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <josef@toxicpanda.com> wrote: > > On 2/24/21 10:47 PM, Neal Gompa wrote: > > On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 2/24/21 9:23 AM, Neal Gompa wrote: > >>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>> > >>>> On 2/22/21 11:03 PM, Neal Gompa wrote: > >>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>> > >>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: > >>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>> > >>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: > >>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>> > >>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: > >>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: > >>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>> Hey all, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure > >>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the > >>>>>>>>>>>>>>>>>>> journal: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in > >>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Can anyone help? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Can you try > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree > >>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some > >>>>>>>>>>>>>>>>>> other magic. Thanks, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Josef > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> No dice... > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 > >>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> build, and then run > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr > >>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind > >>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 > >>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 > >>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>> > >>>>>>>>>>>>> It's better, but still no dice... :( > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. > >>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Nope, I see this in the journal: > >>>>>>>>>>> > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Ok git pull for-neal, rebuild, then run > >>>>>>>>>> > >>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 > >>>>>>>>>> > >>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances > >>>>>>>>>> that the level doesn't match??". Thanks, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> Tried rescue mount again after running that and got a stack trace in > >>>>>>>>> the kernel, detailed in the following attached log. > >>>>>>>> > >>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with > >>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a > >>>>>>>> kernel with a fix for this > >>>>>>>> > >>>>>>>> https://paste.centos.org/view/7b48aaea > >>>>>>>> > >>>>>>>> and see if that gets you further. Thanks, > >>>>>>>> > >>>>>>> > >>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. > >>>>>>> > >>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>>>> Killed > >>>>>>> > >>>>>>> The log from the journal is attached. > >>>>>> > >>>>>> > >>>>>> Ahh crud my bad, this should do it > >>>>>> > >>>>>> https://paste.centos.org/view/ac2e61ef > >>>>>> > >>>>> > >>>>> Patch doesn't apply (note it is patch 667 below): > >>>> > >>>> Ah sorry, should have just sent you an iterative patch. You can take the above > >>>> patch and just delete the hunk from volumes.c as you already have that applied > >>>> and then it'll work. Thanks, > >>>> > >>> > >>> Failed with a weird error...? > >>> > >>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt > >>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>> > >>> Journal log with traceback attached. > >> > >> Last one maybe? > >> > >> https://paste.centos.org/view/80edd6fd > >> > > > > Similar weird failure: > > > > [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > > mount: /mnt: mount(2) system call failed: No such file or directory. > > > > No crash in the journal this time, though: > > > >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options > >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums > >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots > >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time > >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled > >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents > >> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 > >> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed > > > > > > Sorry Neal, you replied when I was in the middle of something and promptly > forgot about it. I figured the fs root was fine, can you do the following so I > can figure out from the error messages what might be wrong > > btrfs check --readonly > btrfs restore -D > btrfs restore -l > It didn't work.. Here's the output: [root@fedora ~]# btrfs check --readonly /dev/sdb3 Opening filesystem to check... ERROR: could not setup extent tree ERROR: cannot open file system [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt WARNING: could not setup extent tree, skipping it Couldn't setup device tree Could not open root, trying backup super parent transid verify failed on 796082176 wanted 888894 found 888896 parent transid verify failed on 796082176 wanted 888894 found 888896 parent transid verify failed on 796082176 wanted 888894 found 888896 Ignoring transid failure WARNING: could not setup extent tree, skipping it Couldn't setup device tree Could not open root, trying backup super ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 Could not open root, trying backup super [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt WARNING: could not setup extent tree, skipping it Couldn't setup device tree Could not open root, trying backup super parent transid verify failed on 796082176 wanted 888894 found 888896 parent transid verify failed on 796082176 wanted 888894 found 888896 parent transid verify failed on 796082176 wanted 888894 found 888896 Ignoring transid failure WARNING: could not setup extent tree, skipping it Couldn't setup device tree Could not open root, trying backup super ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 Could not open root, trying backup super -- 真実はいつも一つ！/ Always, there's only one truth! ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-03-03 19:38 ` Neal Gompa @ 2021-03-04 20:25 ` Josef Bacik 2021-03-04 23:54 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-03-04 20:25 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 3/3/21 2:38 PM, Neal Gompa wrote: > On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 2/24/21 10:47 PM, Neal Gompa wrote: >>> On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>> >>>> On 2/24/21 9:23 AM, Neal Gompa wrote: >>>>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>> >>>>>> On 2/22/21 11:03 PM, Neal Gompa wrote: >>>>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>> >>>>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: >>>>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>> >>>>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: >>>>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: >>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: >>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>> Hey all, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure >>>>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>>>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the >>>>>>>>>>>>>>>>>>>>> journal: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>>>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in >>>>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>>>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>>>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Can anyone help? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Can you try >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree >>>>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some >>>>>>>>>>>>>>>>>>>> other magic. Thanks, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Josef >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> No dice... >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 >>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> build, and then run >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr >>>>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind >>>>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 >>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 >>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It's better, but still no dice... :( >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. >>>>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Nope, I see this in the journal: >>>>>>>>>>>>> >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Ok git pull for-neal, rebuild, then run >>>>>>>>>>>> >>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 >>>>>>>>>>>> >>>>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances >>>>>>>>>>>> that the level doesn't match??". Thanks, >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Tried rescue mount again after running that and got a stack trace in >>>>>>>>>>> the kernel, detailed in the following attached log. >>>>>>>>>> >>>>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with >>>>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a >>>>>>>>>> kernel with a fix for this >>>>>>>>>> >>>>>>>>>> https://paste.centos.org/view/7b48aaea >>>>>>>>>> >>>>>>>>>> and see if that gets you further. Thanks, >>>>>>>>>> >>>>>>>>> >>>>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. >>>>>>>>> >>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>>>>>>>> Killed >>>>>>>>> >>>>>>>>> The log from the journal is attached. >>>>>>>> >>>>>>>> >>>>>>>> Ahh crud my bad, this should do it >>>>>>>> >>>>>>>> https://paste.centos.org/view/ac2e61ef >>>>>>>> >>>>>>> >>>>>>> Patch doesn't apply (note it is patch 667 below): >>>>>> >>>>>> Ah sorry, should have just sent you an iterative patch. You can take the above >>>>>> patch and just delete the hunk from volumes.c as you already have that applied >>>>>> and then it'll work. Thanks, >>>>>> >>>>> >>>>> Failed with a weird error...? >>>>> >>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt >>>>> mount: /mnt: mount(2) system call failed: No such file or directory. >>>>> >>>>> Journal log with traceback attached. >>>> >>>> Last one maybe? >>>> >>>> https://paste.centos.org/view/80edd6fd >>>> >>> >>> Similar weird failure: >>> >>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>> mount: /mnt: mount(2) system call failed: No such file or directory. >>> >>> No crash in the journal this time, though: >>> >>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options >>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums >>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots >>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time >>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled >>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents >>>> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 >>>> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed >>> >>> >> >> Sorry Neal, you replied when I was in the middle of something and promptly >> forgot about it. I figured the fs root was fine, can you do the following so I >> can figure out from the error messages what might be wrong >> >> btrfs check --readonly >> btrfs restore -D >> btrfs restore -l >> > > It didn't work.. Here's the output: > > [root@fedora ~]# btrfs check --readonly /dev/sdb3 > Opening filesystem to check... > ERROR: could not setup extent tree > ERROR: cannot open file system > [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt > WARNING: could not setup extent tree, skipping it > Couldn't setup device tree > Could not open root, trying backup super > parent transid verify failed on 796082176 wanted 888894 found 888896 > parent transid verify failed on 796082176 wanted 888894 found 888896 > parent transid verify failed on 796082176 wanted 888894 found 888896 > Ignoring transid failure > WARNING: could not setup extent tree, skipping it > Couldn't setup device tree > Could not open root, trying backup super > ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > Could not open root, trying backup super > [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt > WARNING: could not setup extent tree, skipping it > Couldn't setup device tree > Could not open root, trying backup super > parent transid verify failed on 796082176 wanted 888894 found 888896 > parent transid verify failed on 796082176 wanted 888894 found 888896 > parent transid verify failed on 796082176 wanted 888894 found 888896 > Ignoring transid failure > WARNING: could not setup extent tree, skipping it > Couldn't setup device tree > Could not open root, trying backup super > ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > Could not open root, trying backup super > > Hmm OK I think we want the neal magic for this one too, but before we go doing that can I get a btrfs inspect-internal -f /dev/whatever so I can make sure I'm not just blindly clobbering something. Thanks, Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-03-04 20:25 ` Josef Bacik @ 2021-03-04 23:54 ` Neal Gompa 2021-03-05 14:12 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-03-04 23:54 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS On Thu, Mar 4, 2021 at 3:25 PM Josef Bacik <josef@toxicpanda.com> wrote: > > On 3/3/21 2:38 PM, Neal Gompa wrote: > > On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 2/24/21 10:47 PM, Neal Gompa wrote: > >>> On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>> > >>>> On 2/24/21 9:23 AM, Neal Gompa wrote: > >>>>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>> > >>>>>> On 2/22/21 11:03 PM, Neal Gompa wrote: > >>>>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>> > >>>>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: > >>>>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>> > >>>>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: > >>>>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: > >>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>> Hey all, > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure > >>>>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>>>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the > >>>>>>>>>>>>>>>>>>>>> journal: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>>>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in > >>>>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>>>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>>>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Can anyone help? > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Can you try > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree > >>>>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some > >>>>>>>>>>>>>>>>>>>> other magic. Thanks, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Josef > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> No dice... > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 > >>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> build, and then run > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr > >>>>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind > >>>>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 > >>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 > >>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> It's better, but still no dice... :( > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. > >>>>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Nope, I see this in the journal: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options > >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums > >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots > >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time > >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Ok git pull for-neal, rebuild, then run > >>>>>>>>>>>> > >>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 > >>>>>>>>>>>> > >>>>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances > >>>>>>>>>>>> that the level doesn't match??". Thanks, > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Tried rescue mount again after running that and got a stack trace in > >>>>>>>>>>> the kernel, detailed in the following attached log. > >>>>>>>>>> > >>>>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with > >>>>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a > >>>>>>>>>> kernel with a fix for this > >>>>>>>>>> > >>>>>>>>>> https://paste.centos.org/view/7b48aaea > >>>>>>>>>> > >>>>>>>>>> and see if that gets you further. Thanks, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. > >>>>>>>>> > >>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>>>>>> Killed > >>>>>>>>> > >>>>>>>>> The log from the journal is attached. > >>>>>>>> > >>>>>>>> > >>>>>>>> Ahh crud my bad, this should do it > >>>>>>>> > >>>>>>>> https://paste.centos.org/view/ac2e61ef > >>>>>>>> > >>>>>>> > >>>>>>> Patch doesn't apply (note it is patch 667 below): > >>>>>> > >>>>>> Ah sorry, should have just sent you an iterative patch. You can take the above > >>>>>> patch and just delete the hunk from volumes.c as you already have that applied > >>>>>> and then it'll work. Thanks, > >>>>>> > >>>>> > >>>>> Failed with a weird error...? > >>>>> > >>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt > >>>>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>>>> > >>>>> Journal log with traceback attached. > >>>> > >>>> Last one maybe? > >>>> > >>>> https://paste.centos.org/view/80edd6fd > >>>> > >>> > >>> Similar weird failure: > >>> > >>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>> > >>> No crash in the journal this time, though: > >>> > >>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options > >>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums > >>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots > >>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time > >>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled > >>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents > >>>> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 > >>>> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed > >>> > >>> > >> > >> Sorry Neal, you replied when I was in the middle of something and promptly > >> forgot about it. I figured the fs root was fine, can you do the following so I > >> can figure out from the error messages what might be wrong > >> > >> btrfs check --readonly > >> btrfs restore -D > >> btrfs restore -l > >> > > > > It didn't work.. Here's the output: > > > > [root@fedora ~]# btrfs check --readonly /dev/sdb3 > > Opening filesystem to check... > > ERROR: could not setup extent tree > > ERROR: cannot open file system > > [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt > > WARNING: could not setup extent tree, skipping it > > Couldn't setup device tree > > Could not open root, trying backup super > > parent transid verify failed on 796082176 wanted 888894 found 888896 > > parent transid verify failed on 796082176 wanted 888894 found 888896 > > parent transid verify failed on 796082176 wanted 888894 found 888896 > > Ignoring transid failure > > WARNING: could not setup extent tree, skipping it > > Couldn't setup device tree > > Could not open root, trying backup super > > ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > > Could not open root, trying backup super > > [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt > > WARNING: could not setup extent tree, skipping it > > Couldn't setup device tree > > Could not open root, trying backup super > > parent transid verify failed on 796082176 wanted 888894 found 888896 > > parent transid verify failed on 796082176 wanted 888894 found 888896 > > parent transid verify failed on 796082176 wanted 888894 found 888896 > > Ignoring transid failure > > WARNING: could not setup extent tree, skipping it > > Couldn't setup device tree > > Could not open root, trying backup super > > ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > > Could not open root, trying backup super > > > > > > Hmm OK I think we want the neal magic for this one too, but before we go doing > that can I get a > > btrfs inspect-internal -f /dev/whatever > > so I can make sure I'm not just blindly clobbering something. Thanks, > Doesn't work, did you mean some other command? [root@fedora ~]# btrfs inspect-internal -f /dev/sdb3 btrfs inspect-internal: unknown token '-f' -- 真実はいつも一つ！/ Always, there's only one truth! ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-03-04 23:54 ` Neal Gompa @ 2021-03-05 14:12 ` Josef Bacik 2021-03-05 14:41 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-03-05 14:12 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 3/4/21 6:54 PM, Neal Gompa wrote: > On Thu, Mar 4, 2021 at 3:25 PM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 3/3/21 2:38 PM, Neal Gompa wrote: >>> On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>> >>>> On 2/24/21 10:47 PM, Neal Gompa wrote: >>>>> On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>> >>>>>> On 2/24/21 9:23 AM, Neal Gompa wrote: >>>>>>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>> >>>>>>>> On 2/22/21 11:03 PM, Neal Gompa wrote: >>>>>>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>> >>>>>>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: >>>>>>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: >>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: >>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>> Hey all, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure >>>>>>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>>>>>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the >>>>>>>>>>>>>>>>>>>>>>> journal: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>>>>>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in >>>>>>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>>>>>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>>>>>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Can anyone help? >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Can you try >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree >>>>>>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some >>>>>>>>>>>>>>>>>>>>>> other magic. Thanks, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Josef >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> No dice... >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 >>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> build, and then run >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr >>>>>>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind >>>>>>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 >>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 >>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> It's better, but still no dice... :( >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. >>>>>>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Nope, I see this in the journal: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ok git pull for-neal, rebuild, then run >>>>>>>>>>>>>> >>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 >>>>>>>>>>>>>> >>>>>>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances >>>>>>>>>>>>>> that the level doesn't match??". Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Tried rescue mount again after running that and got a stack trace in >>>>>>>>>>>>> the kernel, detailed in the following attached log. >>>>>>>>>>>> >>>>>>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with >>>>>>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a >>>>>>>>>>>> kernel with a fix for this >>>>>>>>>>>> >>>>>>>>>>>> https://paste.centos.org/view/7b48aaea >>>>>>>>>>>> >>>>>>>>>>>> and see if that gets you further. Thanks, >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. >>>>>>>>>>> >>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>>>>>>>>>> Killed >>>>>>>>>>> >>>>>>>>>>> The log from the journal is attached. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Ahh crud my bad, this should do it >>>>>>>>>> >>>>>>>>>> https://paste.centos.org/view/ac2e61ef >>>>>>>>>> >>>>>>>>> >>>>>>>>> Patch doesn't apply (note it is patch 667 below): >>>>>>>> >>>>>>>> Ah sorry, should have just sent you an iterative patch. You can take the above >>>>>>>> patch and just delete the hunk from volumes.c as you already have that applied >>>>>>>> and then it'll work. Thanks, >>>>>>>> >>>>>>> >>>>>>> Failed with a weird error...? >>>>>>> >>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt >>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. >>>>>>> >>>>>>> Journal log with traceback attached. >>>>>> >>>>>> Last one maybe? >>>>>> >>>>>> https://paste.centos.org/view/80edd6fd >>>>>> >>>>> >>>>> Similar weird failure: >>>>> >>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>>>> mount: /mnt: mount(2) system call failed: No such file or directory. >>>>> >>>>> No crash in the journal this time, though: >>>>> >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed >>>>> >>>>> >>>> >>>> Sorry Neal, you replied when I was in the middle of something and promptly >>>> forgot about it. I figured the fs root was fine, can you do the following so I >>>> can figure out from the error messages what might be wrong >>>> >>>> btrfs check --readonly >>>> btrfs restore -D >>>> btrfs restore -l >>>> >>> >>> It didn't work.. Here's the output: >>> >>> [root@fedora ~]# btrfs check --readonly /dev/sdb3 >>> Opening filesystem to check... >>> ERROR: could not setup extent tree >>> ERROR: cannot open file system >>> [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt >>> WARNING: could not setup extent tree, skipping it >>> Couldn't setup device tree >>> Could not open root, trying backup super >>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>> Ignoring transid failure >>> WARNING: could not setup extent tree, skipping it >>> Couldn't setup device tree >>> Could not open root, trying backup super >>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 >>> Could not open root, trying backup super >>> [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt >>> WARNING: could not setup extent tree, skipping it >>> Couldn't setup device tree >>> Could not open root, trying backup super >>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>> Ignoring transid failure >>> WARNING: could not setup extent tree, skipping it >>> Couldn't setup device tree >>> Could not open root, trying backup super >>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 >>> Could not open root, trying backup super >>> >>> >> >> Hmm OK I think we want the neal magic for this one too, but before we go doing >> that can I get a >> >> btrfs inspect-internal -f /dev/whatever >> >> so I can make sure I'm not just blindly clobbering something. Thanks, >> > > Doesn't work, did you mean some other command? > > [root@fedora ~]# btrfs inspect-internal -f /dev/sdb3 > btrfs inspect-internal: unknown token '-f' Sigh, sorry, btrfs inspect-internal dump-super -f /dev/sdb3 Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-03-05 14:12 ` Josef Bacik @ 2021-03-05 14:41 ` Neal Gompa 2021-03-05 22:01 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-03-05 14:41 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS [-- Attachment #1: Type: text/plain, Size: 13983 bytes --] On Fri, Mar 5, 2021 at 9:12 AM Josef Bacik <josef@toxicpanda.com> wrote: > > On 3/4/21 6:54 PM, Neal Gompa wrote: > > On Thu, Mar 4, 2021 at 3:25 PM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 3/3/21 2:38 PM, Neal Gompa wrote: > >>> On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>> > >>>> On 2/24/21 10:47 PM, Neal Gompa wrote: > >>>>> On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>> > >>>>>> On 2/24/21 9:23 AM, Neal Gompa wrote: > >>>>>>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>> > >>>>>>>> On 2/22/21 11:03 PM, Neal Gompa wrote: > >>>>>>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>> > >>>>>>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: > >>>>>>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: > >>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>> Hey all, > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure > >>>>>>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>>>>>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the > >>>>>>>>>>>>>>>>>>>>>>> journal: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>>>>>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in > >>>>>>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>>>>>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>>>>>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Can anyone help? > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Can you try > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree > >>>>>>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some > >>>>>>>>>>>>>>>>>>>>>> other magic. Thanks, > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Josef > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> No dice... > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 > >>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> build, and then run > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr > >>>>>>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind > >>>>>>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 > >>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 > >>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> It's better, but still no dice... :( > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. > >>>>>>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Nope, I see this in the journal: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options > >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums > >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots > >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time > >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Ok git pull for-neal, rebuild, then run > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances > >>>>>>>>>>>>>> that the level doesn't match??". Thanks, > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Tried rescue mount again after running that and got a stack trace in > >>>>>>>>>>>>> the kernel, detailed in the following attached log. > >>>>>>>>>>>> > >>>>>>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with > >>>>>>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a > >>>>>>>>>>>> kernel with a fix for this > >>>>>>>>>>>> > >>>>>>>>>>>> https://paste.centos.org/view/7b48aaea > >>>>>>>>>>>> > >>>>>>>>>>>> and see if that gets you further. Thanks, > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. > >>>>>>>>>>> > >>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>>>>>>>> Killed > >>>>>>>>>>> > >>>>>>>>>>> The log from the journal is attached. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Ahh crud my bad, this should do it > >>>>>>>>>> > >>>>>>>>>> https://paste.centos.org/view/ac2e61ef > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> Patch doesn't apply (note it is patch 667 below): > >>>>>>>> > >>>>>>>> Ah sorry, should have just sent you an iterative patch. You can take the above > >>>>>>>> patch and just delete the hunk from volumes.c as you already have that applied > >>>>>>>> and then it'll work. Thanks, > >>>>>>>> > >>>>>>> > >>>>>>> Failed with a weird error...? > >>>>>>> > >>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt > >>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>>>>>> > >>>>>>> Journal log with traceback attached. > >>>>>> > >>>>>> Last one maybe? > >>>>>> > >>>>>> https://paste.centos.org/view/80edd6fd > >>>>>> > >>>>> > >>>>> Similar weird failure: > >>>>> > >>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>>>> > >>>>> No crash in the journal this time, though: > >>>>> > >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options > >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums > >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots > >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time > >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled > >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents > >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 > >>>>>> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed > >>>>> > >>>>> > >>>> > >>>> Sorry Neal, you replied when I was in the middle of something and promptly > >>>> forgot about it. I figured the fs root was fine, can you do the following so I > >>>> can figure out from the error messages what might be wrong > >>>> > >>>> btrfs check --readonly > >>>> btrfs restore -D > >>>> btrfs restore -l > >>>> > >>> > >>> It didn't work.. Here's the output: > >>> > >>> [root@fedora ~]# btrfs check --readonly /dev/sdb3 > >>> Opening filesystem to check... > >>> ERROR: could not setup extent tree > >>> ERROR: cannot open file system > >>> [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt > >>> WARNING: could not setup extent tree, skipping it > >>> Couldn't setup device tree > >>> Could not open root, trying backup super > >>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>> Ignoring transid failure > >>> WARNING: could not setup extent tree, skipping it > >>> Couldn't setup device tree > >>> Could not open root, trying backup super > >>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > >>> Could not open root, trying backup super > >>> [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt > >>> WARNING: could not setup extent tree, skipping it > >>> Couldn't setup device tree > >>> Could not open root, trying backup super > >>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>> Ignoring transid failure > >>> WARNING: could not setup extent tree, skipping it > >>> Couldn't setup device tree > >>> Could not open root, trying backup super > >>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > >>> Could not open root, trying backup super > >>> > >>> > >> > >> Hmm OK I think we want the neal magic for this one too, but before we go doing > >> that can I get a > >> > >> btrfs inspect-internal -f /dev/whatever > >> > >> so I can make sure I'm not just blindly clobbering something. Thanks, > >> > > > > Doesn't work, did you mean some other command? > > > > [root@fedora ~]# btrfs inspect-internal -f /dev/sdb3 > > btrfs inspect-internal: unknown token '-f' > > Sigh, sorry, btrfs inspect-internal dump-super -f /dev/sdb3 > Output from the command attached. -- 真実はいつも一つ！/ Always, there's only one truth! [-- Attachment #2: output-inspect-internal.log --] [-- Type: text/x-log, Size: 3312 bytes --] superblock: bytenr=65536, device=/dev/sdb3 --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0x5fd1da0b [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid f993ffa4-8801-4d57-a087-1c35fd6ece00 metadata_uuid f993ffa4-8801-4d57-a087-1c35fd6ece00 label fedora_f32-ryo-ohki-winvm generation 888895 root 791281664 sys_array_size 129 chunk_root_generation 787284 root_level 2 chunk_root 22036480 chunk_root_level 0 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 263132807168 bytes_used 114928676864 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 1 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x161 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA ) cache_generation 888894 uuid_tree_generation 888894 dev_item.uuid bf50f7f1-bb98-4932-8388-935e4838cb63 dev_item.fsid f993ffa4-8801-4d57-a087-1c35fd6ece00 [match] dev_item.type 0 dev_item.total_bytes 263132807168 dev_item.bytes_used 131021668352 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 1 offset 22020096 dev_uuid bf50f7f1-bb98-4932-8388-935e4838cb63 stripe 1 devid 1 offset 30408704 dev_uuid bf50f7f1-bb98-4932-8388-935e4838cb63 backup_roots[4]: backup 0: backup_tree_root: 791281664 gen: 888893 level: 1 backup_chunk_root: 22036480 gen: 787284 level: 0 backup_extent_root: 790642688 gen: 888893 level: 2 backup_fs_root: 48431104 gen: 757276 level: 0 backup_dev_root: 290979840 gen: 888539 level: 0 backup_csum_root: 789741568 gen: 888893 level: 2 backup_total_bytes: 263132807168 backup_bytes_used: 114928689152 backup_num_devices: 1 backup 1: backup_tree_root: 796082176 gen: 888894 level: 1 backup_chunk_root: 22036480 gen: 787284 level: 0 backup_extent_root: 794951680 gen: 888894 level: 2 backup_fs_root: 48431104 gen: 757276 level: 0 backup_dev_root: 290979840 gen: 888539 level: 0 backup_csum_root: 791461888 gen: 888894 level: 2 backup_total_bytes: 263132807168 backup_bytes_used: 114928676864 backup_num_devices: 1 backup 2: backup_tree_root: 791248896 gen: 888891 level: 1 backup_chunk_root: 22036480 gen: 787284 level: 0 backup_extent_root: 789725184 gen: 888891 level: 2 backup_fs_root: 48431104 gen: 757276 level: 0 backup_dev_root: 290979840 gen: 888539 level: 0 backup_csum_root: 789626880 gen: 888891 level: 2 backup_total_bytes: 263132807168 backup_bytes_used: 114928693248 backup_num_devices: 1 backup 3: backup_tree_root: 795230208 gen: 888892 level: 1 backup_chunk_root: 22036480 gen: 787284 level: 0 backup_extent_root: 793034752 gen: 888892 level: 2 backup_fs_root: 48431104 gen: 757276 level: 0 backup_dev_root: 290979840 gen: 888539 level: 0 backup_csum_root: 789692416 gen: 888892 level: 2 backup_total_bytes: 263132807168 backup_bytes_used: 114928689152 backup_num_devices: 1 ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-03-05 14:41 ` Neal Gompa @ 2021-03-05 22:01 ` Josef Bacik 2021-03-06 1:03 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-03-05 22:01 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 3/5/21 9:41 AM, Neal Gompa wrote: > On Fri, Mar 5, 2021 at 9:12 AM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 3/4/21 6:54 PM, Neal Gompa wrote: >>> On Thu, Mar 4, 2021 at 3:25 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>> >>>> On 3/3/21 2:38 PM, Neal Gompa wrote: >>>>> On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>> >>>>>> On 2/24/21 10:47 PM, Neal Gompa wrote: >>>>>>> On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>> >>>>>>>> On 2/24/21 9:23 AM, Neal Gompa wrote: >>>>>>>>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>> >>>>>>>>>> On 2/22/21 11:03 PM, Neal Gompa wrote: >>>>>>>>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: >>>>>>>>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: >>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>>>> Hey all, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure >>>>>>>>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>>>>>>>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the >>>>>>>>>>>>>>>>>>>>>>>>> journal: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>>>>>>>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in >>>>>>>>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>>>>>>>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>>>>>>>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Can anyone help? >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Can you try >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree >>>>>>>>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some >>>>>>>>>>>>>>>>>>>>>>>> other magic. Thanks, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Josef >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> No dice... >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 >>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> build, and then run >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr >>>>>>>>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind >>>>>>>>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 >>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 >>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> It's better, but still no dice... :( >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. >>>>>>>>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Nope, I see this in the journal: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ok git pull for-neal, rebuild, then run >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances >>>>>>>>>>>>>>>> that the level doesn't match??". Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Tried rescue mount again after running that and got a stack trace in >>>>>>>>>>>>>>> the kernel, detailed in the following attached log. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with >>>>>>>>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a >>>>>>>>>>>>>> kernel with a fix for this >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://paste.centos.org/view/7b48aaea >>>>>>>>>>>>>> >>>>>>>>>>>>>> and see if that gets you further. Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. >>>>>>>>>>>>> >>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>>>>>>>>>>>> Killed >>>>>>>>>>>>> >>>>>>>>>>>>> The log from the journal is attached. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Ahh crud my bad, this should do it >>>>>>>>>>>> >>>>>>>>>>>> https://paste.centos.org/view/ac2e61ef >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Patch doesn't apply (note it is patch 667 below): >>>>>>>>>> >>>>>>>>>> Ah sorry, should have just sent you an iterative patch. You can take the above >>>>>>>>>> patch and just delete the hunk from volumes.c as you already have that applied >>>>>>>>>> and then it'll work. Thanks, >>>>>>>>>> >>>>>>>>> >>>>>>>>> Failed with a weird error...? >>>>>>>>> >>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt >>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. >>>>>>>>> >>>>>>>>> Journal log with traceback attached. >>>>>>>> >>>>>>>> Last one maybe? >>>>>>>> >>>>>>>> https://paste.centos.org/view/80edd6fd >>>>>>>> >>>>>>> >>>>>>> Similar weird failure: >>>>>>> >>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. >>>>>>> >>>>>>> No crash in the journal this time, though: >>>>>>> >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed >>>>>>> >>>>>>> >>>>>> >>>>>> Sorry Neal, you replied when I was in the middle of something and promptly >>>>>> forgot about it. I figured the fs root was fine, can you do the following so I >>>>>> can figure out from the error messages what might be wrong >>>>>> >>>>>> btrfs check --readonly >>>>>> btrfs restore -D >>>>>> btrfs restore -l >>>>>> >>>>> >>>>> It didn't work.. Here's the output: >>>>> >>>>> [root@fedora ~]# btrfs check --readonly /dev/sdb3 >>>>> Opening filesystem to check... >>>>> ERROR: could not setup extent tree >>>>> ERROR: cannot open file system >>>>> [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt >>>>> WARNING: could not setup extent tree, skipping it >>>>> Couldn't setup device tree >>>>> Could not open root, trying backup super >>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>> Ignoring transid failure >>>>> WARNING: could not setup extent tree, skipping it >>>>> Couldn't setup device tree >>>>> Could not open root, trying backup super >>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 >>>>> Could not open root, trying backup super >>>>> [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt >>>>> WARNING: could not setup extent tree, skipping it >>>>> Couldn't setup device tree >>>>> Could not open root, trying backup super >>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>> Ignoring transid failure >>>>> WARNING: could not setup extent tree, skipping it >>>>> Couldn't setup device tree >>>>> Could not open root, trying backup super >>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 >>>>> Could not open root, trying backup super >>>>> >>>>> >>>> >>>> Hmm OK I think we want the neal magic for this one too, but before we go doing >>>> that can I get a >>>> >>>> btrfs inspect-internal -f /dev/whatever >>>> >>>> so I can make sure I'm not just blindly clobbering something. Thanks, >>>> >>> >>> Doesn't work, did you mean some other command? >>> >>> [root@fedora ~]# btrfs inspect-internal -f /dev/sdb3 >>> btrfs inspect-internal: unknown token '-f' >> >> Sigh, sorry, btrfs inspect-internal dump-super -f /dev/sdb3 >> > Ok I've pushed to the for-neal branch in my btrfs-progs, can you pull and make and then run ./btrfs-print-block /dev/sdb3 791281664 and capture everything it prints out? Thanks, Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-03-05 22:01 ` Josef Bacik @ 2021-03-06 1:03 ` Neal Gompa 2021-03-08 18:38 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-03-06 1:03 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS [-- Attachment #1: Type: text/plain, Size: 14835 bytes --] On Fri, Mar 5, 2021 at 5:01 PM Josef Bacik <josef@toxicpanda.com> wrote: > > On 3/5/21 9:41 AM, Neal Gompa wrote: > > On Fri, Mar 5, 2021 at 9:12 AM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 3/4/21 6:54 PM, Neal Gompa wrote: > >>> On Thu, Mar 4, 2021 at 3:25 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>> > >>>> On 3/3/21 2:38 PM, Neal Gompa wrote: > >>>>> On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>> > >>>>>> On 2/24/21 10:47 PM, Neal Gompa wrote: > >>>>>>> On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>> > >>>>>>>> On 2/24/21 9:23 AM, Neal Gompa wrote: > >>>>>>>>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>> > >>>>>>>>>> On 2/22/21 11:03 PM, Neal Gompa wrote: > >>>>>>>>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: > >>>>>>>>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> Hey all, > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure > >>>>>>>>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>>>>>>>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the > >>>>>>>>>>>>>>>>>>>>>>>>> journal: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>>>>>>>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in > >>>>>>>>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>>>>>>>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>>>>>>>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Can anyone help? > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Can you try > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree > >>>>>>>>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some > >>>>>>>>>>>>>>>>>>>>>>>> other magic. Thanks, > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Josef > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> No dice... > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 > >>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> build, and then run > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr > >>>>>>>>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind > >>>>>>>>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 > >>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 > >>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> It's better, but still no dice... :( > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. > >>>>>>>>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Nope, I see this in the journal: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options > >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums > >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots > >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time > >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Ok git pull for-neal, rebuild, then run > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances > >>>>>>>>>>>>>>>> that the level doesn't match??". Thanks, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Tried rescue mount again after running that and got a stack trace in > >>>>>>>>>>>>>>> the kernel, detailed in the following attached log. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with > >>>>>>>>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a > >>>>>>>>>>>>>> kernel with a fix for this > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> https://paste.centos.org/view/7b48aaea > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> and see if that gets you further. Thanks, > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. > >>>>>>>>>>>>> > >>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>>>>>>>>>> Killed > >>>>>>>>>>>>> > >>>>>>>>>>>>> The log from the journal is attached. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Ahh crud my bad, this should do it > >>>>>>>>>>>> > >>>>>>>>>>>> https://paste.centos.org/view/ac2e61ef > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Patch doesn't apply (note it is patch 667 below): > >>>>>>>>>> > >>>>>>>>>> Ah sorry, should have just sent you an iterative patch. You can take the above > >>>>>>>>>> patch and just delete the hunk from volumes.c as you already have that applied > >>>>>>>>>> and then it'll work. Thanks, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> Failed with a weird error...? > >>>>>>>>> > >>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt > >>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>>>>>>>> > >>>>>>>>> Journal log with traceback attached. > >>>>>>>> > >>>>>>>> Last one maybe? > >>>>>>>> > >>>>>>>> https://paste.centos.org/view/80edd6fd > >>>>>>>> > >>>>>>> > >>>>>>> Similar weird failure: > >>>>>>> > >>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>>>>>> > >>>>>>> No crash in the journal this time, though: > >>>>>>> > >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options > >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums > >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots > >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time > >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled > >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents > >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 > >>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> Sorry Neal, you replied when I was in the middle of something and promptly > >>>>>> forgot about it. I figured the fs root was fine, can you do the following so I > >>>>>> can figure out from the error messages what might be wrong > >>>>>> > >>>>>> btrfs check --readonly > >>>>>> btrfs restore -D > >>>>>> btrfs restore -l > >>>>>> > >>>>> > >>>>> It didn't work.. Here's the output: > >>>>> > >>>>> [root@fedora ~]# btrfs check --readonly /dev/sdb3 > >>>>> Opening filesystem to check... > >>>>> ERROR: could not setup extent tree > >>>>> ERROR: cannot open file system > >>>>> [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt > >>>>> WARNING: could not setup extent tree, skipping it > >>>>> Couldn't setup device tree > >>>>> Could not open root, trying backup super > >>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>> Ignoring transid failure > >>>>> WARNING: could not setup extent tree, skipping it > >>>>> Couldn't setup device tree > >>>>> Could not open root, trying backup super > >>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > >>>>> Could not open root, trying backup super > >>>>> [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt > >>>>> WARNING: could not setup extent tree, skipping it > >>>>> Couldn't setup device tree > >>>>> Could not open root, trying backup super > >>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>> Ignoring transid failure > >>>>> WARNING: could not setup extent tree, skipping it > >>>>> Couldn't setup device tree > >>>>> Could not open root, trying backup super > >>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > >>>>> Could not open root, trying backup super > >>>>> > >>>>> > >>>> > >>>> Hmm OK I think we want the neal magic for this one too, but before we go doing > >>>> that can I get a > >>>> > >>>> btrfs inspect-internal -f /dev/whatever > >>>> > >>>> so I can make sure I'm not just blindly clobbering something. Thanks, > >>>> > >>> > >>> Doesn't work, did you mean some other command? > >>> > >>> [root@fedora ~]# btrfs inspect-internal -f /dev/sdb3 > >>> btrfs inspect-internal: unknown token '-f' > >> > >> Sigh, sorry, btrfs inspect-internal dump-super -f /dev/sdb3 > >> > > > > Ok I've pushed to the for-neal branch in my btrfs-progs, can you pull and make > and then run > > ./btrfs-print-block /dev/sdb3 791281664 > > and capture everything it prints out? Thanks, > Here's the output from the command. -- 真実はいつも一つ！/ Always, there's only one truth! [-- Attachment #2: output-print-block.log --] [-- Type: text/x-log, Size: 9271 bytes --] leaf 782893056 items 36 free space 714 generation 888891 owner 401 leaf 782893056 flags 0x1(WRITTEN) backref revision 1 fs uuid f993ffa4-8801-4d57-a087-1c35fd6ece00 chunk uuid 7eff154b-3550-427e-98cb-7300b3d69ab3 item 0 key (203749 EXTENT_DATA 0) itemoff 14847 itemsize 1436 generation 758017 type 0 (inline) inline extent data size 1415 ram_bytes 1415 compression 0 (none) item 1 key (203750 INODE_ITEM 0) itemoff 14687 itemsize 160 generation 758017 transid 758017 size 1417 nbytes 1417 block group 0 mode 100644 links 3 uid 0 gid 0 rdev 0 sequence 25 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555203.80978340 (2021-01-01 21:40:03) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555203.80978340 (2021-01-01 21:40:03) item 2 key (203750 INODE_REF 35767) itemoff 14579 itemsize 108 index 439 namelen 28 name: gb18030.cpython-39.opt-1.pyc index 441 namelen 28 name: gb18030.cpython-39.opt-2.pyc index 443 namelen 22 name: gb18030.cpython-39.pyc item 3 key (203750 XATTR_ITEM 3817753667) itemoff 14506 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 4 key (203750 EXTENT_DATA 0) itemoff 13068 itemsize 1438 generation 758017 type 0 (inline) inline extent data size 1417 ram_bytes 1417 compression 0 (none) item 5 key (203751 INODE_ITEM 0) itemoff 12908 itemsize 160 generation 758017 transid 758017 size 1415 nbytes 1415 block group 0 mode 100644 links 3 uid 0 gid 0 rdev 0 sequence 13 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555203.82978371 (2021-01-01 21:40:03) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555203.81978356 (2021-01-01 21:40:03) item 6 key (203751 INODE_REF 35767) itemoff 12803 itemsize 105 index 445 namelen 27 name: gb2312.cpython-39.opt-1.pyc index 447 namelen 27 name: gb2312.cpython-39.opt-2.pyc index 449 namelen 21 name: gb2312.cpython-39.pyc item 7 key (203751 XATTR_ITEM 3817753667) itemoff 12730 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 8 key (203751 EXTENT_DATA 0) itemoff 11294 itemsize 1436 generation 758017 type 0 (inline) inline extent data size 1415 ram_bytes 1415 compression 0 (none) item 9 key (203752 INODE_ITEM 0) itemoff 11134 itemsize 160 generation 758017 transid 758017 size 1409 nbytes 1409 block group 0 mode 100644 links 3 uid 0 gid 0 rdev 0 sequence 13 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555203.82978371 (2021-01-01 21:40:03) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555203.82978371 (2021-01-01 21:40:03) item 10 key (203752 INODE_REF 35767) itemoff 11038 itemsize 96 index 451 namelen 24 name: gbk.cpython-39.opt-1.pyc index 453 namelen 24 name: gbk.cpython-39.opt-2.pyc index 455 namelen 18 name: gbk.cpython-39.pyc item 11 key (203752 XATTR_ITEM 3817753667) itemoff 10965 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 12 key (203752 EXTENT_DATA 0) itemoff 9535 itemsize 1430 generation 758017 type 0 (inline) inline extent data size 1409 ram_bytes 1409 compression 0 (none) item 13 key (203753 INODE_ITEM 0) itemoff 9375 itemsize 160 generation 758017 transid 758017 size 1407 nbytes 1407 block group 0 mode 100644 links 3 uid 0 gid 0 rdev 0 sequence 13 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555203.84978401 (2021-01-01 21:40:03) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555203.82978371 (2021-01-01 21:40:03) item 14 key (203753 INODE_REF 35767) itemoff 9282 itemsize 93 index 457 namelen 23 name: hz.cpython-39.opt-1.pyc index 459 namelen 23 name: hz.cpython-39.opt-2.pyc index 461 namelen 17 name: hz.cpython-39.pyc item 15 key (203753 XATTR_ITEM 3817753667) itemoff 9209 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 16 key (203753 EXTENT_DATA 0) itemoff 7781 itemsize 1428 generation 758017 type 0 (inline) inline extent data size 1407 ram_bytes 1407 compression 0 (none) item 17 key (203754 INODE_ITEM 0) itemoff 7621 itemsize 160 generation 758017 transid 888891 size 5599 nbytes 8192 block group 0 mode 100644 links 3 uid 0 gid 0 rdev 0 sequence 13 flags 0x0(none) atime 1613320570.860818098 (2021-02-14 11:36:10) ctime 1609555203.85978417 (2021-01-01 21:40:03) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555203.84978401 (2021-01-01 21:40:03) item 18 key (203754 INODE_REF 35767) itemoff 7522 itemsize 99 index 463 namelen 25 name: idna.cpython-39.opt-1.pyc index 465 namelen 25 name: idna.cpython-39.opt-2.pyc index 467 namelen 19 name: idna.cpython-39.pyc item 19 key (203754 XATTR_ITEM 3817753667) itemoff 7449 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 20 key (203754 EXTENT_DATA 0) itemoff 7396 itemsize 53 generation 758017 type 1 (regular) extent data disk byte 5793521664 nr 8192 extent data offset 0 nr 8192 ram 8192 extent compression 0 (none) item 21 key (203755 INODE_ITEM 0) itemoff 7236 itemsize 160 generation 758017 transid 758017 size 1428 nbytes 1428 block group 0 mode 100644 links 3 uid 0 gid 0 rdev 0 sequence 13 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555203.86978432 (2021-01-01 21:40:03) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555203.85978417 (2021-01-01 21:40:03) item 22 key (203755 INODE_REF 35767) itemoff 7119 itemsize 117 index 469 namelen 31 name: iso2022_jp.cpython-39.opt-1.pyc index 471 namelen 31 name: iso2022_jp.cpython-39.opt-2.pyc index 473 namelen 25 name: iso2022_jp.cpython-39.pyc item 23 key (203755 XATTR_ITEM 3817753667) itemoff 7046 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 24 key (203755 EXTENT_DATA 0) itemoff 5597 itemsize 1449 generation 758017 type 0 (inline) inline extent data size 1428 ram_bytes 1428 compression 0 (none) item 25 key (203756 INODE_ITEM 0) itemoff 5437 itemsize 160 generation 758017 transid 758017 size 1432 nbytes 1432 block group 0 mode 100644 links 3 uid 0 gid 0 rdev 0 sequence 13 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555203.87978447 (2021-01-01 21:40:03) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555203.86978432 (2021-01-01 21:40:03) item 26 key (203756 INODE_REF 35767) itemoff 5314 itemsize 123 index 475 namelen 33 name: iso2022_jp_1.cpython-39.opt-1.pyc index 477 namelen 33 name: iso2022_jp_1.cpython-39.opt-2.pyc index 479 namelen 27 name: iso2022_jp_1.cpython-39.pyc item 27 key (203756 XATTR_ITEM 3817753667) itemoff 5241 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 28 key (203756 EXTENT_DATA 0) itemoff 3788 itemsize 1453 generation 758017 type 0 (inline) inline extent data size 1432 ram_bytes 1432 compression 0 (none) item 29 key (203757 INODE_ITEM 0) itemoff 3628 itemsize 160 generation 758017 transid 758017 size 1432 nbytes 1432 block group 0 mode 100644 links 3 uid 0 gid 0 rdev 0 sequence 13 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555203.88978463 (2021-01-01 21:40:03) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555203.87978447 (2021-01-01 21:40:03) item 30 key (203757 INODE_REF 35767) itemoff 3505 itemsize 123 index 481 namelen 33 name: iso2022_jp_2.cpython-39.opt-1.pyc index 483 namelen 33 name: iso2022_jp_2.cpython-39.opt-2.pyc index 485 namelen 27 name: iso2022_jp_2.cpython-39.pyc item 31 key (203757 XATTR_ITEM 3817753667) itemoff 3432 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 item 32 key (203757 EXTENT_DATA 0) itemoff 1979 itemsize 1453 generation 758017 type 0 (inline) inline extent data size 1432 ram_bytes 1432 compression 0 (none) item 33 key (203758 INODE_ITEM 0) itemoff 1819 itemsize 160 generation 758017 transid 758017 size 1438 nbytes 1438 block group 0 mode 100644 links 3 uid 0 gid 0 rdev 0 sequence 13 flags 0x0(none) atime 1607439660.0 (2020-12-08 10:01:00) ctime 1609555203.88978463 (2021-01-01 21:40:03) mtime 1607439660.0 (2020-12-08 10:01:00) otime 1609555203.88978463 (2021-01-01 21:40:03) item 34 key (203758 INODE_REF 35767) itemoff 1687 itemsize 132 index 487 namelen 36 name: iso2022_jp_2004.cpython-39.opt-1.pyc index 489 namelen 36 name: iso2022_jp_2004.cpython-39.opt-2.pyc index 491 namelen 30 name: iso2022_jp_2004.cpython-39.pyc item 35 key (203758 XATTR_ITEM 3817753667) itemoff 1614 itemsize 73 location key (0 UNKNOWN.0 0) type XATTR transid 758017 data_len 27 name_len 16 name: security.selinux data system_u:object_r:lib_t:s0 ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-03-06 1:03 ` Neal Gompa @ 2021-03-08 18:38 ` Josef Bacik 2021-03-08 20:01 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-03-08 18:38 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 3/5/21 8:03 PM, Neal Gompa wrote: > On Fri, Mar 5, 2021 at 5:01 PM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 3/5/21 9:41 AM, Neal Gompa wrote: >>> On Fri, Mar 5, 2021 at 9:12 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>> >>>> On 3/4/21 6:54 PM, Neal Gompa wrote: >>>>> On Thu, Mar 4, 2021 at 3:25 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>> >>>>>> On 3/3/21 2:38 PM, Neal Gompa wrote: >>>>>>> On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>> >>>>>>>> On 2/24/21 10:47 PM, Neal Gompa wrote: >>>>>>>>> On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>> >>>>>>>>>> On 2/24/21 9:23 AM, Neal Gompa wrote: >>>>>>>>>>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 2/22/21 11:03 PM, Neal Gompa wrote: >>>>>>>>>>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: >>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all, >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure >>>>>>>>>>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>>>>>>>>>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the >>>>>>>>>>>>>>>>>>>>>>>>>>> journal: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>>>>>>>>>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in >>>>>>>>>>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>>>>>>>>>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>>>>>>>>>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Can anyone help? >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Can you try >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree >>>>>>>>>>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some >>>>>>>>>>>>>>>>>>>>>>>>>> other magic. Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Josef >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> No dice... >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 >>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> build, and then run >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr >>>>>>>>>>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind >>>>>>>>>>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 >>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 >>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> It's better, but still no dice... :( >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. >>>>>>>>>>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Nope, I see this in the journal: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Ok git pull for-neal, rebuild, then run >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances >>>>>>>>>>>>>>>>>> that the level doesn't match??". Thanks, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Tried rescue mount again after running that and got a stack trace in >>>>>>>>>>>>>>>>> the kernel, detailed in the following attached log. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with >>>>>>>>>>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a >>>>>>>>>>>>>>>> kernel with a fix for this >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://paste.centos.org/view/7b48aaea >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> and see if that gets you further. Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>>>>>>>>>>>>>> Killed >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The log from the journal is attached. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ahh crud my bad, this should do it >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://paste.centos.org/view/ac2e61ef >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Patch doesn't apply (note it is patch 667 below): >>>>>>>>>>>> >>>>>>>>>>>> Ah sorry, should have just sent you an iterative patch. You can take the above >>>>>>>>>>>> patch and just delete the hunk from volumes.c as you already have that applied >>>>>>>>>>>> and then it'll work. Thanks, >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Failed with a weird error...? >>>>>>>>>>> >>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt >>>>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. >>>>>>>>>>> >>>>>>>>>>> Journal log with traceback attached. >>>>>>>>>> >>>>>>>>>> Last one maybe? >>>>>>>>>> >>>>>>>>>> https://paste.centos.org/view/80edd6fd >>>>>>>>>> >>>>>>>>> >>>>>>>>> Similar weird failure: >>>>>>>>> >>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. >>>>>>>>> >>>>>>>>> No crash in the journal this time, though: >>>>>>>>> >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> Sorry Neal, you replied when I was in the middle of something and promptly >>>>>>>> forgot about it. I figured the fs root was fine, can you do the following so I >>>>>>>> can figure out from the error messages what might be wrong >>>>>>>> >>>>>>>> btrfs check --readonly >>>>>>>> btrfs restore -D >>>>>>>> btrfs restore -l >>>>>>>> >>>>>>> >>>>>>> It didn't work.. Here's the output: >>>>>>> >>>>>>> [root@fedora ~]# btrfs check --readonly /dev/sdb3 >>>>>>> Opening filesystem to check... >>>>>>> ERROR: could not setup extent tree >>>>>>> ERROR: cannot open file system >>>>>>> [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt >>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>> Couldn't setup device tree >>>>>>> Could not open root, trying backup super >>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>> Ignoring transid failure >>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>> Couldn't setup device tree >>>>>>> Could not open root, trying backup super >>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 >>>>>>> Could not open root, trying backup super >>>>>>> [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt >>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>> Couldn't setup device tree >>>>>>> Could not open root, trying backup super >>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>> Ignoring transid failure >>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>> Couldn't setup device tree >>>>>>> Could not open root, trying backup super >>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 >>>>>>> Could not open root, trying backup super >>>>>>> >>>>>>> >>>>>> >>>>>> Hmm OK I think we want the neal magic for this one too, but before we go doing >>>>>> that can I get a >>>>>> >>>>>> btrfs inspect-internal -f /dev/whatever >>>>>> >>>>>> so I can make sure I'm not just blindly clobbering something. Thanks, >>>>>> >>>>> >>>>> Doesn't work, did you mean some other command? >>>>> >>>>> [root@fedora ~]# btrfs inspect-internal -f /dev/sdb3 >>>>> btrfs inspect-internal: unknown token '-f' >>>> >>>> Sigh, sorry, btrfs inspect-internal dump-super -f /dev/sdb3 >>>> >>> >> >> Ok I've pushed to the for-neal branch in my btrfs-progs, can you pull and make >> and then run >> >> ./btrfs-print-block /dev/sdb3 791281664 >> >> and capture everything it prints out? Thanks, >> > > Here's the output from the command. > > Hmm looks like the fs is offset a bit, can you do ./btrfs-print-block /dev/sdb3 799670272 also while we're here can I get btrfs-find-root /dev/sdb3 I'd like to see what it thinks. Thanks, Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-03-08 18:38 ` Josef Bacik @ 2021-03-08 20:01 ` Neal Gompa 2021-03-08 22:04 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-03-08 20:01 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS [-- Attachment #1: Type: text/plain, Size: 15938 bytes --] On Mon, Mar 8, 2021 at 1:38 PM Josef Bacik <josef@toxicpanda.com> wrote: > > On 3/5/21 8:03 PM, Neal Gompa wrote: > > On Fri, Mar 5, 2021 at 5:01 PM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 3/5/21 9:41 AM, Neal Gompa wrote: > >>> On Fri, Mar 5, 2021 at 9:12 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>> > >>>> On 3/4/21 6:54 PM, Neal Gompa wrote: > >>>>> On Thu, Mar 4, 2021 at 3:25 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>> > >>>>>> On 3/3/21 2:38 PM, Neal Gompa wrote: > >>>>>>> On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>> > >>>>>>>> On 2/24/21 10:47 PM, Neal Gompa wrote: > >>>>>>>>> On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>> > >>>>>>>>>> On 2/24/21 9:23 AM, Neal Gompa wrote: > >>>>>>>>>>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 2/22/21 11:03 PM, Neal Gompa wrote: > >>>>>>>>>>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all, > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure > >>>>>>>>>>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>>>>>>>>>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the > >>>>>>>>>>>>>>>>>>>>>>>>>>> journal: > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>>>>>>>>>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in > >>>>>>>>>>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>>>>>>>>>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>>>>>>>>>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Can anyone help? > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Can you try > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree > >>>>>>>>>>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some > >>>>>>>>>>>>>>>>>>>>>>>>>> other magic. Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Josef > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> No dice... > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 > >>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> build, and then run > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr > >>>>>>>>>>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind > >>>>>>>>>>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 > >>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 > >>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> It's better, but still no dice... :( > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. > >>>>>>>>>>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Nope, I see this in the journal: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options > >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums > >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots > >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time > >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Ok git pull for-neal, rebuild, then run > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances > >>>>>>>>>>>>>>>>>> that the level doesn't match??". Thanks, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Tried rescue mount again after running that and got a stack trace in > >>>>>>>>>>>>>>>>> the kernel, detailed in the following attached log. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with > >>>>>>>>>>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a > >>>>>>>>>>>>>>>> kernel with a fix for this > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> https://paste.centos.org/view/7b48aaea > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> and see if that gets you further. Thanks, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>>>>>>>>>>>> Killed > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> The log from the journal is attached. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Ahh crud my bad, this should do it > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> https://paste.centos.org/view/ac2e61ef > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Patch doesn't apply (note it is patch 667 below): > >>>>>>>>>>>> > >>>>>>>>>>>> Ah sorry, should have just sent you an iterative patch. You can take the above > >>>>>>>>>>>> patch and just delete the hunk from volumes.c as you already have that applied > >>>>>>>>>>>> and then it'll work. Thanks, > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Failed with a weird error...? > >>>>>>>>>>> > >>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt > >>>>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>>>>>>>>>> > >>>>>>>>>>> Journal log with traceback attached. > >>>>>>>>>> > >>>>>>>>>> Last one maybe? > >>>>>>>>>> > >>>>>>>>>> https://paste.centos.org/view/80edd6fd > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> Similar weird failure: > >>>>>>>>> > >>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>>>>>>>> > >>>>>>>>> No crash in the journal this time, though: > >>>>>>>>> > >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options > >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums > >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots > >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time > >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled > >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents > >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 > >>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> Sorry Neal, you replied when I was in the middle of something and promptly > >>>>>>>> forgot about it. I figured the fs root was fine, can you do the following so I > >>>>>>>> can figure out from the error messages what might be wrong > >>>>>>>> > >>>>>>>> btrfs check --readonly > >>>>>>>> btrfs restore -D > >>>>>>>> btrfs restore -l > >>>>>>>> > >>>>>>> > >>>>>>> It didn't work.. Here's the output: > >>>>>>> > >>>>>>> [root@fedora ~]# btrfs check --readonly /dev/sdb3 > >>>>>>> Opening filesystem to check... > >>>>>>> ERROR: could not setup extent tree > >>>>>>> ERROR: cannot open file system > >>>>>>> [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt > >>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>> Couldn't setup device tree > >>>>>>> Could not open root, trying backup super > >>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>> Ignoring transid failure > >>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>> Couldn't setup device tree > >>>>>>> Could not open root, trying backup super > >>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > >>>>>>> Could not open root, trying backup super > >>>>>>> [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt > >>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>> Couldn't setup device tree > >>>>>>> Could not open root, trying backup super > >>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>> Ignoring transid failure > >>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>> Couldn't setup device tree > >>>>>>> Could not open root, trying backup super > >>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > >>>>>>> Could not open root, trying backup super > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> Hmm OK I think we want the neal magic for this one too, but before we go doing > >>>>>> that can I get a > >>>>>> > >>>>>> btrfs inspect-internal -f /dev/whatever > >>>>>> > >>>>>> so I can make sure I'm not just blindly clobbering something. Thanks, > >>>>>> > >>>>> > >>>>> Doesn't work, did you mean some other command? > >>>>> > >>>>> [root@fedora ~]# btrfs inspect-internal -f /dev/sdb3 > >>>>> btrfs inspect-internal: unknown token '-f' > >>>> > >>>> Sigh, sorry, btrfs inspect-internal dump-super -f /dev/sdb3 > >>>> > >>> > >> > >> Ok I've pushed to the for-neal branch in my btrfs-progs, can you pull and make > >> and then run > >> > >> ./btrfs-print-block /dev/sdb3 791281664 > >> > >> and capture everything it prints out? Thanks, > >> > > > > Here's the output from the command. > > > > > > Hmm looks like the fs is offset a bit, can you do > > ./btrfs-print-block /dev/sdb3 799670272 > This command caused my session to crash, but I do have a log of what was captured before it crashed and attached it. > also while we're here can I get > > btrfs-find-root /dev/sdb3 > This ran successfully and I've attached the output. > I'd like to see what it thinks. Thanks, > > Josef -- 真実はいつも一つ！/ Always, there's only one truth! [-- Attachment #2: output-print-block2.log.zst --] [-- Type: application/zstd, Size: 14836 bytes --] [-- Attachment #3: output-find-root.log --] [-- Type: text/x-log, Size: 147220 bytes --] WARNING: could not setup extent tree, skipping it Couldn't setup device tree Superblock thinks the generation is 888895 Superblock thinks the level is 2 Well block 37994627072(gen: 889467 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37954945024(gen: 889459 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37937823744(gen: 889457 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37925486592(gen: 889452 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37923012608(gen: 889451 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37921538048(gen: 889450 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37920636928(gen: 889449 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37917949952(gen: 889447 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37914935296(gen: 889441 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37914099712(gen: 889438 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37900812288(gen: 889435 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37858836480(gen: 889434 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37845696512(gen: 889433 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37849038848(gen: 889431 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37844484096(gen: 889430 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37833408512(gen: 889387 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37796315136(gen: 889383 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37795397632(gen: 889382 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37796970496(gen: 889381 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37795332096(gen: 889381 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37784502272(gen: 889359 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37786042368(gen: 889354 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37777768448(gen: 889348 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37772230656(gen: 889340 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37768151040(gen: 889339 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37768871936(gen: 889338 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37761728512(gen: 889323 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37757976576(gen: 889322 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37732876288(gen: 889320 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38117031936(gen: 889319 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37699387392(gen: 889315 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37694193664(gen: 889313 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37698928640(gen: 889310 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37696503808(gen: 889308 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37694111744(gen: 889301 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37693800448(gen: 889299 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37687296000(gen: 889294 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37683970048(gen: 889292 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37683953664(gen: 889292 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37681938432(gen: 889290 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37676957696(gen: 889285 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37672697856(gen: 889284 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37673009152(gen: 889283 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37670010880(gen: 889282 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37670780928(gen: 889281 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37670223872(gen: 889280 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37670076416(gen: 889279 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37667127296(gen: 889277 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37667930112(gen: 889276 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37666799616(gen: 889272 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37664915456(gen: 889270 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37664833536(gen: 889268 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37664358400(gen: 889266 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37660475392(gen: 889262 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37658181632(gen: 889260 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37658132480(gen: 889259 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37657051136(gen: 889257 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37656985600(gen: 889255 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37653872640(gen: 889253 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37655429120(gen: 889252 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37654642688(gen: 889251 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37654495232(gen: 889250 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37650153472(gen: 889234 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37647400960(gen: 889231 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37646958592(gen: 889230 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37645303808(gen: 889229 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37704105984(gen: 889227 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37696258048(gen: 889226 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1100775424(gen: 889219 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1098088448(gen: 889218 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1096646656(gen: 889217 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1095385088(gen: 889216 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1094500352(gen: 889215 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1093353472(gen: 889214 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1090633728(gen: 889213 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1089175552(gen: 889212 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1085456384(gen: 889210 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1083523072(gen: 889209 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1082114048(gen: 889208 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1077198848(gen: 889206 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1076150272(gen: 889205 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1075265536(gen: 889204 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1073020928(gen: 889203 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1070874624(gen: 889200 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1068761088(gen: 889199 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1065041920(gen: 889196 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1062584320(gen: 889195 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1057783808(gen: 889191 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1055997952(gen: 889188 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1050886144(gen: 889187 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1047330816(gen: 889185 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1046085632(gen: 889184 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1045561344(gen: 889183 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1044758528(gen: 889182 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1043333120(gen: 889181 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1042415616(gen: 889180 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1039155200(gen: 889177 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1037500416(gen: 889176 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1036206080(gen: 889175 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1035141120(gen: 889174 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1034092544(gen: 889173 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1033895936(gen: 889171 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1031897088(gen: 889169 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1028292608(gen: 889168 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1027358720(gen: 889167 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1026621440(gen: 889166 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1025441792(gen: 889165 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1024163840(gen: 889162 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1017921536(gen: 889155 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1017085952(gen: 889154 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1016152064(gen: 889153 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1015693312(gen: 889152 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1015103488(gen: 889151 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1013612544(gen: 889150 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1009008640(gen: 889143 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1008091136(gen: 889142 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1007337472(gen: 889140 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1006305280(gen: 889139 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1006043136(gen: 889137 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1004027904(gen: 889136 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1002717184(gen: 889135 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1002094592(gen: 889134 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1000194048(gen: 889131 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 990658560(gen: 889118 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 989364224(gen: 889117 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 988463104(gen: 889116 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 987889664(gen: 889115 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 987463680(gen: 889114 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 986808320(gen: 889113 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 985858048(gen: 889112 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 985104384(gen: 889111 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 984039424(gen: 889110 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 983252992(gen: 889109 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 982433792(gen: 889108 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 981270528(gen: 889107 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 980647936(gen: 889106 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 980844544(gen: 889105 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 978993152(gen: 889103 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 977649664(gen: 889102 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 976896000(gen: 889101 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 975011840(gen: 889098 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 973848576(gen: 889097 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 972521472(gen: 889096 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 971702272(gen: 889095 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 970866688(gen: 889094 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 967262208(gen: 889093 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 967507968(gen: 889092 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 962887680(gen: 889089 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 962265088(gen: 889088 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 961626112(gen: 889087 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 961069056(gen: 889086 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 960544768(gen: 889085 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 960118784(gen: 889084 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 959709184(gen: 889083 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 959315968(gen: 889082 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 958824448(gen: 889081 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 958316544(gen: 889080 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 958054400(gen: 889079 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 957251584(gen: 889077 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 956710912(gen: 889076 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 955138048(gen: 889073 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 954466304(gen: 889072 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 953696256(gen: 889071 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 953008128(gen: 889070 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 953188352(gen: 889069 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 952090624(gen: 889067 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 951681024(gen: 889066 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 951189504(gen: 889065 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 950779904(gen: 889064 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 950288384(gen: 889063 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 949944320(gen: 889062 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 949567488(gen: 889061 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 949174272(gen: 889060 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 948666368(gen: 889059 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 948305920(gen: 889058 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 947486720(gen: 889057 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 947503104(gen: 889056 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 946618368(gen: 889054 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 945373184(gen: 889051 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 944832512(gen: 889050 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 944226304(gen: 889049 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 943718400(gen: 889048 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 943652864(gen: 889047 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 939524096(gen: 889038 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 938868736(gen: 889037 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 938426368(gen: 889036 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 937721856(gen: 889035 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 937115648(gen: 889034 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 936525824(gen: 889033 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 936099840(gen: 889032 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 935526400(gen: 889031 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 935002112(gen: 889030 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 934543360(gen: 889029 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 934051840(gen: 889028 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 933478400(gen: 889027 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 933019648(gen: 889026 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 932577280(gen: 889025 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 932151296(gen: 889024 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 931414016(gen: 889023 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 930988032(gen: 889022 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 930316288(gen: 889021 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 929693696(gen: 889020 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 929071104(gen: 889018 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 928284672(gen: 889017 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 926924800(gen: 889014 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 926105600(gen: 889013 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 925646848(gen: 889012 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 925089792(gen: 889011 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 924532736(gen: 889010 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 924073984(gen: 889009 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 923664384(gen: 889008 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 922943488(gen: 889007 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 922451968(gen: 889006 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 921665536(gen: 889005 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 921206784(gen: 889004 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 920698880(gen: 889003 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 920190976(gen: 889002 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 919715840(gen: 889001 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 919207936(gen: 889000 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 918667264(gen: 888999 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 918077440(gen: 888998 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 917504000(gen: 888997 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 916946944(gen: 888996 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 916504576(gen: 888995 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 916062208(gen: 888994 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 915521536(gen: 888993 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 914997248(gen: 888992 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 914292736(gen: 888991 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 913555456(gen: 888990 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 912998400(gen: 888989 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 912474112(gen: 888988 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 911867904(gen: 888987 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 911278080(gen: 888986 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 910622720(gen: 888985 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 910082048(gen: 888984 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 909295616(gen: 888983 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 908328960(gen: 888982 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 908361728(gen: 888981 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 908165120(gen: 888980 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 906969088(gen: 888979 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 906248192(gen: 888978 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 905625600(gen: 888977 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 905216000(gen: 888976 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 904757248(gen: 888975 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 904101888(gen: 888974 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 903168000(gen: 888973 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 902807552(gen: 888972 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 902004736(gen: 888971 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 900300800(gen: 888968 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 899661824(gen: 888967 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 899088384(gen: 888966 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 898236416(gen: 888965 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 897630208(gen: 888964 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 896876544(gen: 888963 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 896319488(gen: 888962 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 895746048(gen: 888961 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 895156224(gen: 888960 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 894468096(gen: 888959 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 893681664(gen: 888958 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 893026304(gen: 888957 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 892469248(gen: 888956 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 891879424(gen: 888955 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 891273216(gen: 888954 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 890601472(gen: 888953 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 889995264(gen: 888952 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 889159680(gen: 888951 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 888684544(gen: 888950 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 888209408(gen: 888949 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 887668736(gen: 888948 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 886980608(gen: 888947 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 886259712(gen: 888946 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 885424128(gen: 888945 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 884834304(gen: 888944 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 884326400(gen: 888943 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 883589120(gen: 888942 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 882049024(gen: 888940 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 881606656(gen: 888939 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 878444544(gen: 888934 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 877674496(gen: 888933 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 876937216(gen: 888932 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 876101632(gen: 888931 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 876052480(gen: 888930 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 876511232(gen: 888925 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 868941824(gen: 888922 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 871235584(gen: 888921 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 867794944(gen: 888920 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 867008512(gen: 888916 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 856965120(gen: 888915 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 848281600(gen: 888910 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 852869120(gen: 888909 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 840744960(gen: 888904 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 821952512(gen: 888903 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 834453504(gen: 888902 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 821739520(gen: 888901 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 812318720(gen: 888900 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 792395776(gen: 888895 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 780648448(gen: 888890 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 772538368(gen: 888881 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 767524864(gen: 888875 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 776192000(gen: 888872 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 754024448(gen: 888861 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 750649344(gen: 888857 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 749060096(gen: 888855 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 753090560(gen: 888852 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 700088320(gen: 888851 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 700399616(gen: 888850 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 699678720(gen: 888849 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 698777600(gen: 888848 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 697647104(gen: 888847 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 696713216(gen: 888846 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 695943168(gen: 888845 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 565149696(gen: 888843 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 563249152(gen: 888842 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 555352064(gen: 888837 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 563888128(gen: 888836 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 554401792(gen: 888835 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 552796160(gen: 888834 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 550584320(gen: 888825 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 546684928(gen: 888825 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 549994496(gen: 888824 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 549502976(gen: 888823 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 549044224(gen: 888822 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 544309248(gen: 888821 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 548601856(gen: 888820 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 543784960(gen: 888812 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 537739264(gen: 888804 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 539017216(gen: 888801 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 536395776(gen: 888798 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 536100864(gen: 888797 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 534511616(gen: 888794 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 532217856(gen: 888793 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 529678336(gen: 888791 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 528351232(gen: 888790 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 520388608(gen: 888789 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 520306688(gen: 888788 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 517603328(gen: 888786 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 515604480(gen: 888785 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 513835008(gen: 888781 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 509247488(gen: 888779 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 506691584(gen: 888777 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 503808000(gen: 888776 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 503119872(gen: 888776 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 499433472(gen: 888774 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 497516544(gen: 888773 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 497287168(gen: 888772 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 482656256(gen: 888762 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 465272832(gen: 888761 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 478887936(gen: 888758 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 475299840(gen: 888754 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 472219648(gen: 888753 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 464502784(gen: 888745 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 451837952(gen: 888744 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 454410240(gen: 888742 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 453656576(gen: 888741 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 451641344(gen: 888740 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 450445312(gen: 888739 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 450199552(gen: 888738 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 449314816(gen: 888737 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 442417152(gen: 888734 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 442073088(gen: 888733 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 441778176(gen: 888733 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 440172544(gen: 888731 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 439975936(gen: 888730 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 439500800(gen: 888729 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 438861824(gen: 888728 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 434012160(gen: 888727 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 432242688(gen: 888726 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 428507136(gen: 888725 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 427081728(gen: 888718 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 426098688(gen: 888716 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 425082880(gen: 888713 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 424886272(gen: 888712 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 424312832(gen: 888710 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 423215104(gen: 888708 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 423051264(gen: 888707 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 421920768(gen: 888706 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 421445632(gen: 888705 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 420823040(gen: 888704 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 420413440(gen: 888703 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 412991488(gen: 888701 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 412172288(gen: 888700 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 411123712(gen: 888699 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 295632896(gen: 888698 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 287145984(gen: 888697 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 291520512(gen: 888682 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 270336000(gen: 888669 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 265617408(gen: 888668 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 301989888(gen: 888667 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 262553600(gen: 888662 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 260079616(gen: 888662 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 191774720(gen: 888589 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 212729856(gen: 888586 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 212697088(gen: 888582 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 212615168(gen: 888582 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 211877888(gen: 888581 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 175357952(gen: 888576 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 164724736(gen: 888575 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 142688256(gen: 888568 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 127188992(gen: 888566 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 325976064(gen: 888564 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 325206016(gen: 888563 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 321339392(gen: 888561 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 314720256(gen: 888560 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 337690624(gen: 888556 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 323289088(gen: 888552 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 316227584(gen: 888550 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 307200000(gen: 888545 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 301498368(gen: 888542 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 693501952(gen: 888490 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 684441600(gen: 888489 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 680968192(gen: 888488 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 677806080(gen: 888487 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 678756352(gen: 888483 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 668008448(gen: 888477 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 667680768(gen: 888474 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 662929408(gen: 888472 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 661209088(gen: 888471 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 660586496(gen: 888470 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 660111360(gen: 888468 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 659226624(gen: 888467 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 658046976(gen: 888466 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 657375232(gen: 888465 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 652836864(gen: 888451 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 642990080(gen: 888442 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 637829120(gen: 888440 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 635928576(gen: 888439 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 634994688(gen: 888438 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 634011648(gen: 888436 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 632782848(gen: 888435 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 632455168(gen: 888434 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 631898112(gen: 888433 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 629604352(gen: 888426 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 623542272(gen: 888425 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 625000448(gen: 888424 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 622395392(gen: 888423 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 614154240(gen: 888412 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 572686336(gen: 888401 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 408502272(gen: 888322 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 400850944(gen: 888321 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 396689408(gen: 888318 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 392364032(gen: 888312 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 350978048(gen: 888282 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 348880896(gen: 888279 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 323846144(gen: 888261 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 308199424(gen: 888252 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 73036677120(gen: 888052 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 73025568768(gen: 888051 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72900820992(gen: 888039 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72902885376(gen: 888038 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72891711488(gen: 888036 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72888745984(gen: 888035 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72885518336(gen: 888034 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72883978240(gen: 888033 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72883142656(gen: 888032 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72881848320(gen: 888030 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72878604288(gen: 888029 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72876769280(gen: 888028 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72876392448(gen: 888027 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72875753472(gen: 888026 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72870346752(gen: 888022 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72865447936(gen: 888021 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72841641984(gen: 888002 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72840708096(gen: 888001 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72839151616(gen: 888000 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72838496256(gen: 887999 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72836874240(gen: 887995 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72830763008(gen: 887992 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72823242752(gen: 887987 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72820932608(gen: 887985 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72821047296(gen: 887984 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72819884032(gen: 887982 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72819277824(gen: 887981 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72818933760(gen: 887979 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72817836032(gen: 887978 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72811659264(gen: 887972 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72811069440(gen: 887971 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72809676800(gen: 887969 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72798322688(gen: 887958 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72788033536(gen: 887949 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72786558976(gen: 887948 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72786837504(gen: 887947 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72784789504(gen: 887945 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72783986688(gen: 887944 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72780627968(gen: 887939 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72779710464(gen: 887938 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72780398592(gen: 887937 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72779481088(gen: 887936 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72777613312(gen: 887935 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72776876032(gen: 887934 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72774909952(gen: 887933 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72775057408(gen: 887932 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72769847296(gen: 887928 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72768929792(gen: 887927 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72769601536(gen: 887926 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72768471040(gen: 887925 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72766685184(gen: 887924 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72765341696(gen: 887923 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72764260352(gen: 887922 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72762949632(gen: 887921 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72762359808(gen: 887919 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72760508416(gen: 887918 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72757215232(gen: 887915 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72756314112(gen: 887913 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72753496064(gen: 887912 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72751497216(gen: 887909 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72750399488(gen: 887908 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72748810240(gen: 887907 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72747466752(gen: 887906 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72746876928(gen: 887905 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72746598400(gen: 887904 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72745353216(gen: 887903 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72744386560(gen: 887902 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72743813120(gen: 887900 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72740585472(gen: 887897 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72738619392(gen: 887896 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72735899648(gen: 887895 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72733392896(gen: 887894 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72733769728(gen: 887893 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72731607040(gen: 887891 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72729509888(gen: 887890 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72728182784(gen: 887889 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72727117824(gen: 887888 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72726659072(gen: 887887 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72723726336(gen: 887884 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72723103744(gen: 887882 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72721498112(gen: 887880 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72719073280(gen: 887879 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72717910016(gen: 887878 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72714698752(gen: 887875 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72713994240(gen: 887874 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72713338880(gen: 887873 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72712159232(gen: 887872 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72711356416(gen: 887871 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72711028736(gen: 887870 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72710193152(gen: 887868 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72708882432(gen: 887867 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72708112384(gen: 887866 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72691892224(gen: 887847 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72690237440(gen: 887846 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72689664000(gen: 887844 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72687534080(gen: 887843 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72665808896(gen: 887820 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72659779584(gen: 887817 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72658010112(gen: 887815 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72654979072(gen: 887814 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72654061568(gen: 887813 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72653242368(gen: 887812 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72651522048(gen: 887809 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72650178560(gen: 887808 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72647426048(gen: 887807 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72644755456(gen: 887804 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72641757184(gen: 887803 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72638349312(gen: 887802 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72636907520(gen: 887801 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72636121088(gen: 887800 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72635891712(gen: 887799 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72634990592(gen: 887797 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72634515456(gen: 887796 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72634007552(gen: 887795 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72633516032(gen: 887794 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72627912704(gen: 887781 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72626929664(gen: 887780 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72621867008(gen: 887774 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72619098112(gen: 887769 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72617082880(gen: 887766 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72616460288(gen: 887765 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72608268288(gen: 887757 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72607170560(gen: 887756 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72606777344(gen: 887754 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72605433856(gen: 887753 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72596504576(gen: 887741 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72531165184(gen: 887695 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72521859072(gen: 887692 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72519843840(gen: 887690 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72512716800(gen: 887684 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72510734336(gen: 887683 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72507555840(gen: 887682 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72504754176(gen: 887681 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72489811968(gen: 887676 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72476426240(gen: 887667 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72472576000(gen: 887665 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72468365312(gen: 887659 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72467644416(gen: 887658 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72466841600(gen: 887656 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72457715712(gen: 887646 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72454094848(gen: 887644 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72452571136(gen: 887643 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72451899392(gen: 887642 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72450424832(gen: 887639 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72449212416(gen: 887638 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72448475136(gen: 887637 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72447754240(gen: 887636 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72447148032(gen: 887635 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72445886464(gen: 887634 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72444985344(gen: 887633 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72441593856(gen: 887629 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72437547008(gen: 887626 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72429076480(gen: 887622 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72428830720(gen: 887621 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72427159552(gen: 887620 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72425766912(gen: 887619 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72424718336(gen: 887616 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72423342080(gen: 887615 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72421244928(gen: 887614 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72419983360(gen: 887613 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72418787328(gen: 887612 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72417705984(gen: 887611 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72417067008(gen: 887610 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72416804864(gen: 887609 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72415674368(gen: 887607 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72414380032(gen: 887604 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72410316800(gen: 887599 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72409792512(gen: 887598 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72408858624(gen: 887596 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72398651392(gen: 887586 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72396111872(gen: 887585 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72393277440(gen: 887582 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72392261632(gen: 887580 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72390197248(gen: 887579 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72389509120(gen: 887578 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72389214208(gen: 887578 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72389148672(gen: 887578 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72373403648(gen: 887577 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72372568064(gen: 887576 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72370077696(gen: 887574 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72368816128(gen: 887573 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72369307648(gen: 887572 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72367587328(gen: 887571 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72367063040(gen: 887570 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72366850048(gen: 887569 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72366292992(gen: 887568 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72362704896(gen: 887561 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72361459712(gen: 887560 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72360525824(gen: 887559 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72360329216(gen: 887558 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72358199296(gen: 887554 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72357625856(gen: 887553 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72357920768(gen: 887550 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72351531008(gen: 887546 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72350695424(gen: 887545 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72348958720(gen: 887542 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72347959296(gen: 887541 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72347090944(gen: 887540 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72346206208(gen: 887538 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72343781376(gen: 887537 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72342618112(gen: 887536 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72340766720(gen: 887533 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72340357120(gen: 887532 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72339521536(gen: 887531 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72338292736(gen: 887530 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72337686528(gen: 887529 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72336113664(gen: 887527 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72335294464(gen: 887526 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72334540800(gen: 887525 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72333950976(gen: 887524 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72333295616(gen: 887523 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72332771328(gen: 887522 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72332115968(gen: 887521 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72331689984(gen: 887520 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72331952128(gen: 887519 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72330641408(gen: 887518 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72330149888(gen: 887517 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72325382144(gen: 887509 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72323186688(gen: 887506 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72321368064(gen: 887503 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72319795200(gen: 887502 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72318681088(gen: 887499 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72318533632(gen: 887497 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72318484480(gen: 887497 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72316059648(gen: 887496 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72315781120(gen: 887494 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72315207680(gen: 887493 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72314355712(gen: 887492 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72313241600(gen: 887491 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72313077760(gen: 887490 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72312799232(gen: 887490 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72312029184(gen: 887489 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72311013376(gen: 887488 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72310784000(gen: 887487 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72310538240(gen: 887486 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72310259712(gen: 887485 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72309506048(gen: 887483 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72309227520(gen: 887482 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72308162560(gen: 887480 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72307752960(gen: 887479 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72305524736(gen: 887478 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72304738304(gen: 887477 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72298872832(gen: 887476 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72298414080(gen: 887475 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72298102784(gen: 887475 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72297398272(gen: 887474 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72297086976(gen: 887473 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72296873984(gen: 887473 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72296071168(gen: 887472 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72293531648(gen: 887471 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72288239616(gen: 887470 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72286289920(gen: 887469 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72285683712(gen: 887468 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72284962816(gen: 887467 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72284372992(gen: 887466 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72284078080(gen: 887465 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72283537408(gen: 887464 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72282931200(gen: 887463 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72282783744(gen: 887462 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72282570752(gen: 887462 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72279965696(gen: 887461 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72277573632(gen: 887460 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72278016000(gen: 887459 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72277786624(gen: 887458 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72277098496(gen: 887457 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72276819968(gen: 887457 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72276017152(gen: 887456 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72275279872(gen: 887455 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72274542592(gen: 887454 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72273805312(gen: 887453 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72273330176(gen: 887452 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72272674816(gen: 887451 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72271986688(gen: 887450 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72270905344(gen: 887449 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72270200832(gen: 887448 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72269414400(gen: 887447 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72268529664(gen: 887446 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72267841536(gen: 887445 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72267087872(gen: 887444 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72266301440(gen: 887443 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72265236480(gen: 887442 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72262631424(gen: 887441 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72103772160(gen: 887440 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72102461440(gen: 887426 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72097775616(gen: 887419 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72101281792(gen: 887418 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72100626432(gen: 887417 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72100233216(gen: 887416 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72097366016(gen: 887415 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72098758656(gen: 887414 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72092696576(gen: 887412 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72090959872(gen: 887410 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72094105600(gen: 887409 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72092975104(gen: 887408 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72092073984(gen: 887407 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72091713536(gen: 887407 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72090910720(gen: 887404 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72090320896(gen: 887404 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72084799488(gen: 887403 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72088518656(gen: 887402 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72087486464(gen: 887400 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72085159936(gen: 887398 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72085127168(gen: 887397 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72083210240(gen: 887393 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72077131776(gen: 887392 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72079589376(gen: 887387 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72069021696(gen: 887380 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72068169728(gen: 887376 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72066580480(gen: 887374 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72064335872(gen: 887373 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72063942656(gen: 887372 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72062533632(gen: 887370 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72061911040(gen: 887369 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72061747200(gen: 887368 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72060878848(gen: 887367 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72060682240(gen: 887366 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72059633664(gen: 887362 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72058142720(gen: 887361 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72053325824(gen: 887360 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72058322944(gen: 887353 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72056504320(gen: 887350 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72053440512(gen: 887339 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72052342784(gen: 887331 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72051621888(gen: 887330 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72050737152(gen: 887328 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72039743488(gen: 887327 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72019492864(gen: 887316 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72019034112(gen: 887311 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72018788352(gen: 887310 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72017838080(gen: 887308 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72017690624(gen: 887307 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72017199104(gen: 887307 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72016691200(gen: 887306 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72016117760(gen: 887305 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72008269824(gen: 887303 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72007090176(gen: 887302 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72005468160(gen: 887301 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72003682304(gen: 887300 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72002633728(gen: 887299 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72001568768(gen: 887299 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72387035136(gen: 887121 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72387100672(gen: 887098 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72168947712(gen: 887085 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72385478656(gen: 887035 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72166293504(gen: 887021 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72151302144(gen: 887021 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72151269376(gen: 887021 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72001208320(gen: 887006 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72000569344(gen: 887006 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71967506432(gen: 887004 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71934017536(gen: 886982 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71933149184(gen: 886981 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71932706816(gen: 886979 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71921369088(gen: 886972 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71920205824(gen: 886971 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71919173632(gen: 886969 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71918616576(gen: 886968 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71914242048(gen: 886966 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71913340928(gen: 886965 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71907016704(gen: 886963 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71905607680(gen: 886962 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71904444416(gen: 886960 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71898677248(gen: 886957 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71897792512(gen: 886956 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71896989696(gen: 886955 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71894515712(gen: 886953 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71888814080(gen: 886950 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71888994304(gen: 886949 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71887257600(gen: 886947 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71886323712(gen: 886945 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71886094336(gen: 886943 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71884505088(gen: 886941 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71880720384(gen: 886935 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71879507968(gen: 886933 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71878574080(gen: 886932 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71872184320(gen: 886930 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71871021056(gen: 886929 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71866187776(gen: 886927 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71865647104(gen: 886926 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71865073664(gen: 886925 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71864270848(gen: 886924 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71863943168(gen: 886923 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71863435264(gen: 886922 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71862796288(gen: 886921 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71861993472(gen: 886920 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71861452800(gen: 886919 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71861075968(gen: 886918 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71860535296(gen: 886917 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71860027392(gen: 886916 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71859355648(gen: 886915 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71858651136(gen: 886914 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71858028544(gen: 886913 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71857078272(gen: 886912 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71855521792(gen: 886908 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71854129152(gen: 886906 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71852851200(gen: 886904 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71852179456(gen: 886903 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71851573248(gen: 886902 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71851180032(gen: 886901 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71850557440(gen: 886900 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71850180608(gen: 886899 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71849607168(gen: 886898 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71848984576(gen: 886897 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71848198144(gen: 886896 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71846592512(gen: 886894 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71840677888(gen: 886891 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71839219712(gen: 886889 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71829700608(gen: 886888 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71828750336(gen: 886887 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71825653760(gen: 886885 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71823884288(gen: 886884 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71821262848(gen: 886879 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71820525568(gen: 886878 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71820017664(gen: 886877 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71819444224(gen: 886876 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71818592256(gen: 886875 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71818379264(gen: 886874 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71813398528(gen: 886865 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71812513792(gen: 886864 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71808876544(gen: 886862 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71807500288(gen: 886861 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71806369792(gen: 886859 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71803994112(gen: 886855 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71803453440(gen: 886854 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71802994688(gen: 886853 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71802322944(gen: 886852 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71801339904(gen: 886851 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71800127488(gen: 886849 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71799095296(gen: 886846 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71798030336(gen: 886845 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71798685696(gen: 886844 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71796785152(gen: 886843 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71788920832(gen: 886829 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71788085248(gen: 886828 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71787266048(gen: 886827 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71784103936(gen: 886825 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71781466112(gen: 886822 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71769128960(gen: 886818 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71767703552(gen: 886817 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71764525056(gen: 886814 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71763738624(gen: 886813 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71759052800(gen: 886811 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71758610432(gen: 886809 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71754366976(gen: 886808 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71753531392(gen: 886807 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71753662464(gen: 886806 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71748648960(gen: 886803 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71747944448(gen: 886802 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71742619648(gen: 886800 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71742095360(gen: 886799 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71737589760(gen: 886797 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71736967168(gen: 886796 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71734427648(gen: 886794 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71721861120(gen: 886786 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71712260096(gen: 886783 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71708524544(gen: 886780 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71701889024(gen: 886777 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71700905984(gen: 886776 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71697924096(gen: 886774 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71697219584(gen: 886773 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71692779520(gen: 886771 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71691796480(gen: 886770 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71688421376(gen: 886768 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71687192576(gen: 886767 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71684997120(gen: 886765 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71683833856(gen: 886764 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71679344640(gen: 886762 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71675576320(gen: 886758 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71673954304(gen: 886757 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71669530624(gen: 886755 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71666991104(gen: 886754 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71665336320(gen: 886753 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71665221632(gen: 886751 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71664041984(gen: 886750 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71662305280(gen: 886747 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71662026752(gen: 886746 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71660077056(gen: 886742 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71659175936(gen: 886741 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71657488384(gen: 886738 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71656095744(gen: 886735 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71651966976(gen: 886730 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71651262464(gen: 886729 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71647477760(gen: 886724 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71647166464(gen: 886722 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71645888512(gen: 886721 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71645298688(gen: 886720 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71639760896(gen: 886712 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71639482368(gen: 886711 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71638384640(gen: 886709 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71637434368(gen: 886708 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71635795968(gen: 886706 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71635369984(gen: 886705 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71626997760(gen: 886694 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71625277440(gen: 886692 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71623147520(gen: 886691 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71619690496(gen: 886687 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71615987712(gen: 886685 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71612530688(gen: 886684 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71610859520(gen: 886683 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71601291264(gen: 886681 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71599407104(gen: 886680 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71588659200(gen: 886678 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71587299328(gen: 886676 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71584186368(gen: 886675 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71578091520(gen: 886673 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71575306240(gen: 886671 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71560822784(gen: 886668 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71549534208(gen: 886666 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71548043264(gen: 886665 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71545438208(gen: 886664 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71544438784(gen: 886663 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71538917376(gen: 886661 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71532625920(gen: 886659 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71530332160(gen: 886658 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71524843520(gen: 886656 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71522844672(gen: 886655 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71516782592(gen: 886653 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71506132992(gen: 886650 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71505526784(gen: 886649 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71501447168(gen: 886647 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71498629120(gen: 886645 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71497957376(gen: 886644 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71491436544(gen: 886642 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71485374464(gen: 886639 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71476723712(gen: 886636 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71457505280(gen: 886630 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71450279936(gen: 886627 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71448969216(gen: 886626 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71442661376(gen: 886624 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71441432576(gen: 886623 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71435763712(gen: 886621 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71435386880(gen: 886620 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71434338304(gen: 886619 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71426965504(gen: 886618 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71420313600(gen: 886615 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71411859456(gen: 886612 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71411482624(gen: 886611 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71410860032(gen: 886610 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71406075904(gen: 886609 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71399227392(gen: 886606 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71398703104(gen: 886605 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71394689024(gen: 886603 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71387381760(gen: 886600 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71386218496(gen: 886599 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71381630976(gen: 886597 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71375077376(gen: 886594 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71374651392(gen: 886593 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71370113024(gen: 886591 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71369605120(gen: 886590 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71364919296(gen: 886588 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71362265088(gen: 886587 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71354220544(gen: 886585 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71353794560(gen: 886584 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71348568064(gen: 886582 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71347699712(gen: 886581 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71340916736(gen: 886579 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71340163072(gen: 886578 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71334035456(gen: 886576 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71333232640(gen: 886575 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71328120832(gen: 886573 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71327563776(gen: 886572 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71321501696(gen: 886570 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71320600576(gen: 886569 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71314718720(gen: 886567 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71307657216(gen: 886564 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71307362304(gen: 886563 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71306772480(gen: 886562 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71299907584(gen: 886561 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71298498560(gen: 886560 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71292370944(gen: 886558 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71291863040(gen: 886556 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71285882880(gen: 886555 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71281115136(gen: 886552 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71277264896(gen: 886550 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71269269504(gen: 886547 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71262322688(gen: 886544 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71262158848(gen: 886542 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71255801856(gen: 886541 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71249936384(gen: 886538 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71244562432(gen: 886535 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71244103680(gen: 886534 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71240646656(gen: 886532 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71239565312(gen: 886531 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71233880064(gen: 886529 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71232634880(gen: 886528 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71233683456(gen: 886527 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71226048512(gen: 886526 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71225393152(gen: 886525 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71218905088(gen: 886523 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71218380800(gen: 886522 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71211679744(gen: 886520 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71211057152(gen: 886519 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71204274176(gen: 886517 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71203848192(gen: 886516 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71199571968(gen: 886514 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71192674304(gen: 886511 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71187152896(gen: 886508 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71176011776(gen: 886505 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71175471104(gen: 886504 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71174782976(gen: 886503 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71168917504(gen: 886502 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71168360448(gen: 886501 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71167623168(gen: 886500 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71159250944(gen: 886499 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71157547008(gen: 886498 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71150911488(gen: 886496 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71150338048(gen: 886495 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71144816640(gen: 886493 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71143096320(gen: 886492 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71138050048(gen: 886490 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71136886784(gen: 886489 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71122157568(gen: 886484 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71115915264(gen: 886481 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71114768384(gen: 886480 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71110836224(gen: 886478 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71109197824(gen: 886477 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71105183744(gen: 886475 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71104217088(gen: 886474 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71099875328(gen: 886472 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71086899200(gen: 886467 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71077019648(gen: 886463 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71086047232(gen: 886461 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71085621248(gen: 886461 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71072858112(gen: 886460 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71078445056(gen: 886458 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71071301632(gen: 886457 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71072350208(gen: 886456 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71051182080(gen: 886454 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71048101888(gen: 886453 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71053524992(gen: 886452 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71045775360(gen: 886451 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71042924544(gen: 886450 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71046397952(gen: 886449 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71040319488(gen: 886448 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71040385024(gen: 886446 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71040106496(gen: 886446 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71034060800(gen: 886445 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71032569856(gen: 886444 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71030407168(gen: 886443 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71008632832(gen: 886442 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 70978830336(gen: 886439 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71004602368(gen: 886438 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 70918668288(gen: 886229 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 70916800512(gen: 886228 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 70914129920(gen: 886227 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 70911148032(gen: 886226 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 70906085376(gen: 886225 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 70901022720(gen: 886224 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38684213248(gen: 886223 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38682411008(gen: 886222 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38679347200(gen: 886221 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38677233664(gen: 886219 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38676643840(gen: 886218 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38674726912(gen: 886216 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38673530880(gen: 886215 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38671810560(gen: 886212 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38670794752(gen: 886211 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38669975552(gen: 886210 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38669729792(gen: 886209 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38667730944(gen: 886207 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38666862592(gen: 886206 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38668992512(gen: 886205 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38663798784(gen: 886203 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38663995392(gen: 886202 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38663192576(gen: 886201 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38662242304(gen: 886200 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38661799936(gen: 886199 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38661013504(gen: 886198 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38660440064(gen: 886197 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38659768320(gen: 886196 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38657982464(gen: 886194 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38657179648(gen: 886193 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38655639552(gen: 886190 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38654509056(gen: 886189 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38652919808(gen: 886188 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38652395520(gen: 886186 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38651576320(gen: 886185 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38650757120(gen: 886184 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38649102336(gen: 886183 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38647644160(gen: 886181 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38643548160(gen: 886180 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38642647040(gen: 886179 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38642057216(gen: 886177 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38640173056(gen: 886176 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38636929024(gen: 886173 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38632079360(gen: 886172 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38631342080(gen: 886171 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38630670336(gen: 886170 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38629572608(gen: 886168 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38628065280(gen: 886167 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38627098624(gen: 886166 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38626394112(gen: 886165 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38625624064(gen: 886164 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38624362496(gen: 886163 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38623428608(gen: 886162 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38622216192(gen: 886161 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38620987392(gen: 886159 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38617776128(gen: 886155 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38615646208(gen: 886153 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38613221376(gen: 886151 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38612107264(gen: 886150 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38613172224(gen: 886149 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38608715776(gen: 886147 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38606520320(gen: 886144 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38602014720(gen: 886143 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38590676992(gen: 886138 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38584483840(gen: 886137 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38580092928(gen: 886134 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38589726720(gen: 886132 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38551748608(gen: 886129 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38164135936(gen: 886128 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38095683584(gen: 886127 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38095192064(gen: 886125 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38036160512(gen: 886114 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38015516672(gen: 886099 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38014304256(gen: 886098 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 1076871168(gen: 885846 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 972357632(gen: 885747 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 70922190848(gen: 885266 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72907603968(gen: 883908 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72906440704(gen: 883907 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72386936832(gen: 883284 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72384937984(gen: 883283 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72178663424(gen: 883244 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72194572288(gen: 883209 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72165801984(gen: 883204 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72178171904(gen: 883152 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72177631232(gen: 883152 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72174567424(gen: 883150 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72167112704(gen: 883148 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72158216192(gen: 883147 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72115781632(gen: 883143 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72198537216(gen: 883129 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72182693888(gen: 883124 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72173617152(gen: 883120 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72165015552(gen: 883116 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72151564288(gen: 883112 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72151482368(gen: 883112 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72148467712(gen: 883111 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72122138624(gen: 883109 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71991934976(gen: 883092 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71433338880(gen: 882748 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71414628352(gen: 882718 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71335870464(gen: 882602 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71291355136(gen: 882583 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71285194752(gen: 882573 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71268057088(gen: 882546 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71262994432(gen: 882539 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71254802432(gen: 882528 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71254769664(gen: 882528 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 72237662208(gen: 880671 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 102547456(gen: 877652 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 91275264(gen: 877604 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 85049344(gen: 877603 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 77299712(gen: 877602 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71987003392(gen: 876446 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71978696704(gen: 876272 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 70901202944(gen: 875021 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71986806784(gen: 873068 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71973830656(gen: 873015 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 70902857728(gen: 871819 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 70902038528(gen: 871614 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71983906816(gen: 869174 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71980892160(gen: 869174 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71980335104(gen: 869174 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71978352640(gen: 869113 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71975878656(gen: 869111 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 71971602432(gen: 869108 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 48742400(gen: 865947 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 45449216(gen: 865946 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 70900776960(gen: 844439 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 48611328(gen: 828122 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 57262080(gen: 827948 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 70900498432(gen: 825355 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37628575744(gen: 793293 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37626085376(gen: 784021 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37626626048(gen: 778888 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37625970688(gen: 778850 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37620334592(gen: 699360 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37622530048(gen: 696132 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37621415936(gen: 694443 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37621022720(gen: 694443 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37621547008(gen: 694423 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37620023296(gen: 694422 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37617778688(gen: 692016 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37617762304(gen: 692016 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 70898909184(gen: 687752 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37617369088(gen: 686282 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37617156096(gen: 686281 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37616959488(gen: 686281 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37616631808(gen: 684294 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 39763968(gen: 680257 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37615697920(gen: 633848 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37615304704(gen: 572294 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37614682112(gen: 572293 level: 1) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 38486016(gen: 568101 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 Well block 37614419968(gen: 561836 level: 0) seems good, but generation/level doesn't match, want gen: 888895 level: 2 ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-03-08 20:01 ` Neal Gompa @ 2021-03-08 22:04 ` Josef Bacik 2021-03-09 1:12 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-03-08 22:04 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 3/8/21 3:01 PM, Neal Gompa wrote: > On Mon, Mar 8, 2021 at 1:38 PM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 3/5/21 8:03 PM, Neal Gompa wrote: >>> On Fri, Mar 5, 2021 at 5:01 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>> >>>> On 3/5/21 9:41 AM, Neal Gompa wrote: >>>>> On Fri, Mar 5, 2021 at 9:12 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>> >>>>>> On 3/4/21 6:54 PM, Neal Gompa wrote: >>>>>>> On Thu, Mar 4, 2021 at 3:25 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>> >>>>>>>> On 3/3/21 2:38 PM, Neal Gompa wrote: >>>>>>>>> On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>> >>>>>>>>>> On 2/24/21 10:47 PM, Neal Gompa wrote: >>>>>>>>>>> On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 2/24/21 9:23 AM, Neal Gompa wrote: >>>>>>>>>>>>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2/22/21 11:03 PM, Neal Gompa wrote: >>>>>>>>>>>>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure >>>>>>>>>>>>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> journal: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>>>>>>>>>>>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>>>>>>>>>>>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can anyone help? >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you try >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree >>>>>>>>>>>>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some >>>>>>>>>>>>>>>>>>>>>>>>>>>> other magic. Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Josef >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> No dice... >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 >>>>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> build, and then run >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr >>>>>>>>>>>>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind >>>>>>>>>>>>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 >>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 >>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> It's better, but still no dice... :( >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. >>>>>>>>>>>>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Nope, I see this in the journal: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Ok git pull for-neal, rebuild, then run >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances >>>>>>>>>>>>>>>>>>>> that the level doesn't match??". Thanks, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Tried rescue mount again after running that and got a stack trace in >>>>>>>>>>>>>>>>>>> the kernel, detailed in the following attached log. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with >>>>>>>>>>>>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a >>>>>>>>>>>>>>>>>> kernel with a fix for this >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> https://paste.centos.org/view/7b48aaea >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> and see if that gets you further. Thanks, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>>>>>>>>>>>>>>>> Killed >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The log from the journal is attached. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ahh crud my bad, this should do it >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://paste.centos.org/view/ac2e61ef >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Patch doesn't apply (note it is patch 667 below): >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ah sorry, should have just sent you an iterative patch. You can take the above >>>>>>>>>>>>>> patch and just delete the hunk from volumes.c as you already have that applied >>>>>>>>>>>>>> and then it'll work. Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Failed with a weird error...? >>>>>>>>>>>>> >>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt >>>>>>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. >>>>>>>>>>>>> >>>>>>>>>>>>> Journal log with traceback attached. >>>>>>>>>>>> >>>>>>>>>>>> Last one maybe? >>>>>>>>>>>> >>>>>>>>>>>> https://paste.centos.org/view/80edd6fd >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Similar weird failure: >>>>>>>>>>> >>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>>>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. >>>>>>>>>>> >>>>>>>>>>> No crash in the journal this time, though: >>>>>>>>>>> >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Sorry Neal, you replied when I was in the middle of something and promptly >>>>>>>>>> forgot about it. I figured the fs root was fine, can you do the following so I >>>>>>>>>> can figure out from the error messages what might be wrong >>>>>>>>>> >>>>>>>>>> btrfs check --readonly >>>>>>>>>> btrfs restore -D >>>>>>>>>> btrfs restore -l >>>>>>>>>> >>>>>>>>> >>>>>>>>> It didn't work.. Here's the output: >>>>>>>>> >>>>>>>>> [root@fedora ~]# btrfs check --readonly /dev/sdb3 >>>>>>>>> Opening filesystem to check... >>>>>>>>> ERROR: could not setup extent tree >>>>>>>>> ERROR: cannot open file system >>>>>>>>> [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt >>>>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>>>> Couldn't setup device tree >>>>>>>>> Could not open root, trying backup super >>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>> Ignoring transid failure >>>>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>>>> Couldn't setup device tree >>>>>>>>> Could not open root, trying backup super >>>>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 >>>>>>>>> Could not open root, trying backup super >>>>>>>>> [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt >>>>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>>>> Couldn't setup device tree >>>>>>>>> Could not open root, trying backup super >>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>> Ignoring transid failure >>>>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>>>> Couldn't setup device tree >>>>>>>>> Could not open root, trying backup super >>>>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 >>>>>>>>> Could not open root, trying backup super >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> Hmm OK I think we want the neal magic for this one too, but before we go doing >>>>>>>> that can I get a >>>>>>>> >>>>>>>> btrfs inspect-internal -f /dev/whatever >>>>>>>> >>>>>>>> so I can make sure I'm not just blindly clobbering something. Thanks, >>>>>>>> >>>>>>> >>>>>>> Doesn't work, did you mean some other command? >>>>>>> >>>>>>> [root@fedora ~]# btrfs inspect-internal -f /dev/sdb3 >>>>>>> btrfs inspect-internal: unknown token '-f' >>>>>> >>>>>> Sigh, sorry, btrfs inspect-internal dump-super -f /dev/sdb3 >>>>>> >>>>> >>>> >>>> Ok I've pushed to the for-neal branch in my btrfs-progs, can you pull and make >>>> and then run >>>> >>>> ./btrfs-print-block /dev/sdb3 791281664 >>>> >>>> and capture everything it prints out? Thanks, >>>> >>> >>> Here's the output from the command. >>> >>> >> >> Hmm looks like the fs is offset a bit, can you do >> >> ./btrfs-print-block /dev/sdb3 799670272 >> > > This command caused my session to crash, but I do have a log of what > was captured before it crashed and attached it. > >> also while we're here can I get >> >> btrfs-find-root /dev/sdb3 >> > > This ran successfully and I've attached the output. > Ok we're going to try this again, and if it doesn't work it looks like your chunk root is ok, so I'll rig something up to make the translation work right, but for now lets do ./btrfs-print-block /dev/sdb3 792395776 Thanks, Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-03-08 22:04 ` Josef Bacik @ 2021-03-09 1:12 ` Neal Gompa 2021-03-09 19:04 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-03-09 1:12 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS [-- Attachment #1: Type: text/plain, Size: 16923 bytes --] On Mon, Mar 8, 2021 at 5:04 PM Josef Bacik <josef@toxicpanda.com> wrote: > > On 3/8/21 3:01 PM, Neal Gompa wrote: > > On Mon, Mar 8, 2021 at 1:38 PM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 3/5/21 8:03 PM, Neal Gompa wrote: > >>> On Fri, Mar 5, 2021 at 5:01 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>> > >>>> On 3/5/21 9:41 AM, Neal Gompa wrote: > >>>>> On Fri, Mar 5, 2021 at 9:12 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>> > >>>>>> On 3/4/21 6:54 PM, Neal Gompa wrote: > >>>>>>> On Thu, Mar 4, 2021 at 3:25 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>> > >>>>>>>> On 3/3/21 2:38 PM, Neal Gompa wrote: > >>>>>>>>> On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>> > >>>>>>>>>> On 2/24/21 10:47 PM, Neal Gompa wrote: > >>>>>>>>>>> On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 2/24/21 9:23 AM, Neal Gompa wrote: > >>>>>>>>>>>>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 2/22/21 11:03 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> journal: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can anyone help? > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you try > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree > >>>>>>>>>>>>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some > >>>>>>>>>>>>>>>>>>>>>>>>>>>> other magic. Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Josef > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> No dice... > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 > >>>>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> build, and then run > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr > >>>>>>>>>>>>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind > >>>>>>>>>>>>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 > >>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 > >>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> It's better, but still no dice... :( > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. > >>>>>>>>>>>>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Nope, I see this in the journal: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options > >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums > >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots > >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time > >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Ok git pull for-neal, rebuild, then run > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances > >>>>>>>>>>>>>>>>>>>> that the level doesn't match??". Thanks, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Tried rescue mount again after running that and got a stack trace in > >>>>>>>>>>>>>>>>>>> the kernel, detailed in the following attached log. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with > >>>>>>>>>>>>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a > >>>>>>>>>>>>>>>>>> kernel with a fix for this > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> https://paste.centos.org/view/7b48aaea > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> and see if that gets you further. Thanks, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>>>>>>>>>>>>>> Killed > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> The log from the journal is attached. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Ahh crud my bad, this should do it > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> https://paste.centos.org/view/ac2e61ef > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Patch doesn't apply (note it is patch 667 below): > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Ah sorry, should have just sent you an iterative patch. You can take the above > >>>>>>>>>>>>>> patch and just delete the hunk from volumes.c as you already have that applied > >>>>>>>>>>>>>> and then it'll work. Thanks, > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Failed with a weird error...? > >>>>>>>>>>>>> > >>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt > >>>>>>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Journal log with traceback attached. > >>>>>>>>>>>> > >>>>>>>>>>>> Last one maybe? > >>>>>>>>>>>> > >>>>>>>>>>>> https://paste.centos.org/view/80edd6fd > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Similar weird failure: > >>>>>>>>>>> > >>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>>>>>>>>>> > >>>>>>>>>>> No crash in the journal this time, though: > >>>>>>>>>>> > >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options > >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums > >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots > >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time > >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled > >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents > >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 > >>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Sorry Neal, you replied when I was in the middle of something and promptly > >>>>>>>>>> forgot about it. I figured the fs root was fine, can you do the following so I > >>>>>>>>>> can figure out from the error messages what might be wrong > >>>>>>>>>> > >>>>>>>>>> btrfs check --readonly > >>>>>>>>>> btrfs restore -D > >>>>>>>>>> btrfs restore -l > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> It didn't work.. Here's the output: > >>>>>>>>> > >>>>>>>>> [root@fedora ~]# btrfs check --readonly /dev/sdb3 > >>>>>>>>> Opening filesystem to check... > >>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>> ERROR: cannot open file system > >>>>>>>>> [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt > >>>>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>>>> Couldn't setup device tree > >>>>>>>>> Could not open root, trying backup super > >>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>> Ignoring transid failure > >>>>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>>>> Couldn't setup device tree > >>>>>>>>> Could not open root, trying backup super > >>>>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > >>>>>>>>> Could not open root, trying backup super > >>>>>>>>> [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt > >>>>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>>>> Couldn't setup device tree > >>>>>>>>> Could not open root, trying backup super > >>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>> Ignoring transid failure > >>>>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>>>> Couldn't setup device tree > >>>>>>>>> Could not open root, trying backup super > >>>>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > >>>>>>>>> Could not open root, trying backup super > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> Hmm OK I think we want the neal magic for this one too, but before we go doing > >>>>>>>> that can I get a > >>>>>>>> > >>>>>>>> btrfs inspect-internal -f /dev/whatever > >>>>>>>> > >>>>>>>> so I can make sure I'm not just blindly clobbering something. Thanks, > >>>>>>>> > >>>>>>> > >>>>>>> Doesn't work, did you mean some other command? > >>>>>>> > >>>>>>> [root@fedora ~]# btrfs inspect-internal -f /dev/sdb3 > >>>>>>> btrfs inspect-internal: unknown token '-f' > >>>>>> > >>>>>> Sigh, sorry, btrfs inspect-internal dump-super -f /dev/sdb3 > >>>>>> > >>>>> > >>>> > >>>> Ok I've pushed to the for-neal branch in my btrfs-progs, can you pull and make > >>>> and then run > >>>> > >>>> ./btrfs-print-block /dev/sdb3 791281664 > >>>> > >>>> and capture everything it prints out? Thanks, > >>>> > >>> > >>> Here's the output from the command. > >>> > >>> > >> > >> Hmm looks like the fs is offset a bit, can you do > >> > >> ./btrfs-print-block /dev/sdb3 799670272 > >> > > > > This command caused my session to crash, but I do have a log of what > > was captured before it crashed and attached it. > > > >> also while we're here can I get > >> > >> btrfs-find-root /dev/sdb3 > >> > > > > This ran successfully and I've attached the output. > > > > Ok we're going to try this again, and if it doesn't work it looks like your > chunk root is ok, so I'll rig something up to make the translation work right, > but for now lets do > > ./btrfs-print-block /dev/sdb3 792395776 > I've attached the output from that command, which did run successfully. -- 真実はいつも一つ！/ Always, there's only one truth! [-- Attachment #2: output-print-block3.log --] [-- Type: text/x-log, Size: 12461 bytes --] leaf 784007168 items 53 free space 7414 generation 757286 owner 401 leaf 784007168 flags 0x1(WRITTEN) backref revision 1 fs uuid f993ffa4-8801-4d57-a087-1c35fd6ece00 chunk uuid 7eff154b-3550-427e-98cb-7300b3d69ab3 item 0 key (147591 XATTR_ITEM 3817753667) itemoff 16207 itemsize 76 location key (0 UNKNOWN.0 0) type XATTR transid 757286 data_len 30 name_len 16 name: security.selinux data system_u:object_r:locale_t:s0 item 1 key (147591 EXTENT_DATA 0) itemoff 16154 itemsize 53 generation 757286 type 1 (regular) extent data disk byte 5798076416 nr 8192 extent data offset 0 nr 8192 ram 8192 extent compression 0 (none) item 2 key (147592 INODE_ITEM 0) itemoff 15994 itemsize 160 generation 757286 transid 757286 size 6248 nbytes 8192 block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x0(none) atime 1609554203.226158833 (2021-01-01 21:23:23) ctime 1609554203.226158833 (2021-01-01 21:23:23) mtime 1603831043.0 (2020-10-27 16:37:23) otime 1609554203.226158833 (2021-01-01 21:23:23) item 3 key (147592 INODE_REF 146020) itemoff 15966 itemsize 28 index 498 namelen 18 name: kcoreaddons5_qt.qm item 4 key (147592 XATTR_ITEM 3817753667) itemoff 15890 itemsize 76 location key (0 UNKNOWN.0 0) type XATTR transid 757286 data_len 30 name_len 16 name: security.selinux data system_u:object_r:locale_t:s0 item 5 key (147592 EXTENT_DATA 0) itemoff 15837 itemsize 53 generation 757286 type 1 (regular) extent data disk byte 5798084608 nr 8192 extent data offset 0 nr 8192 ram 8192 extent compression 0 (none) item 6 key (147593 INODE_ITEM 0) itemoff 15677 itemsize 160 generation 757286 transid 757286 size 1222 nbytes 1222 block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x0(none) atime 1609554203.226158833 (2021-01-01 21:23:23) ctime 1609554203.226158833 (2021-01-01 21:23:23) mtime 1603831030.0 (2020-10-27 16:37:10) otime 1609554203.226158833 (2021-01-01 21:23:23) item 7 key (147593 INODE_REF 146020) itemoff 15649 itemsize 28 index 500 namelen 18 name: kdbusaddons5_qt.qm item 8 key (147593 XATTR_ITEM 3817753667) itemoff 15573 itemsize 76 location key (0 UNKNOWN.0 0) type XATTR transid 757286 data_len 30 name_len 16 name: security.selinux data system_u:object_r:locale_t:s0 item 9 key (147593 EXTENT_DATA 0) itemoff 14330 itemsize 1243 generation 757286 type 0 (inline) inline extent data size 1222 ram_bytes 1222 compression 0 (none) item 10 key (147594 INODE_ITEM 0) itemoff 14170 itemsize 160 generation 757286 transid 757286 size 5830 nbytes 8192 block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x0(none) atime 1609554203.226158833 (2021-01-01 21:23:23) ctime 1609554203.226158833 (2021-01-01 21:23:23) mtime 1603831044.0 (2020-10-27 16:37:24) otime 1609554203.226158833 (2021-01-01 21:23:23) item 11 key (147594 INODE_REF 146020) itemoff 14139 itemsize 31 index 502 namelen 21 name: kde5_xml_mimetypes.qm item 12 key (147594 XATTR_ITEM 3817753667) itemoff 14063 itemsize 76 location key (0 UNKNOWN.0 0) type XATTR transid 757286 data_len 30 name_len 16 name: security.selinux data system_u:object_r:locale_t:s0 item 13 key (147594 EXTENT_DATA 0) itemoff 14010 itemsize 53 generation 757286 type 1 (regular) extent data disk byte 5798092800 nr 8192 extent data offset 0 nr 8192 ram 8192 extent compression 0 (none) item 14 key (147595 INODE_ITEM 0) itemoff 13850 itemsize 160 generation 757286 transid 757286 size 8349 nbytes 12288 block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x0(none) atime 1609554203.226158833 (2021-01-01 21:23:23) ctime 1609554203.226158833 (2021-01-01 21:23:23) mtime 1606453733.0 (2020-11-27 00:08:53) otime 1609554203.226158833 (2021-01-01 21:23:23) item 15 key (147595 INODE_REF 146020) itemoff 13824 itemsize 26 index 504 namelen 16 name: kdeclarative5.mo item 16 key (147595 XATTR_ITEM 3817753667) itemoff 13748 itemsize 76 location key (0 UNKNOWN.0 0) type XATTR transid 757286 data_len 30 name_len 16 name: security.selinux data system_u:object_r:locale_t:s0 item 17 key (147595 EXTENT_DATA 0) itemoff 13695 itemsize 53 generation 757286 type 1 (regular) extent data disk byte 5798100992 nr 12288 extent data offset 0 nr 12288 ram 12288 extent compression 0 (none) item 18 key (147596 INODE_ITEM 0) itemoff 13535 itemsize 160 generation 757286 transid 757286 size 2601 nbytes 4096 block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x0(none) atime 1609554203.226158833 (2021-01-01 21:23:23) ctime 1609554203.226158833 (2021-01-01 21:23:23) mtime 1601653667.0 (2020-10-02 11:47:47) otime 1609554203.226158833 (2021-01-01 21:23:23) item 19 key (147596 INODE_REF 146020) itemoff 13508 itemsize 27 index 506 namelen 17 name: kdeconnect-app.mo item 20 key (147596 XATTR_ITEM 3817753667) itemoff 13432 itemsize 76 location key (0 UNKNOWN.0 0) type XATTR transid 757286 data_len 30 name_len 16 name: security.selinux data system_u:object_r:locale_t:s0 item 21 key (147596 EXTENT_DATA 0) itemoff 13379 itemsize 53 generation 757286 type 1 (regular) extent data disk byte 5798113280 nr 4096 extent data offset 0 nr 4096 ram 4096 extent compression 0 (none) item 22 key (147597 INODE_ITEM 0) itemoff 13219 itemsize 160 generation 757286 transid 757286 size 5693 nbytes 8192 block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x0(none) atime 1609554203.226158833 (2021-01-01 21:23:23) ctime 1609554203.227159091 (2021-01-01 21:23:23) mtime 1601653667.0 (2020-10-02 11:47:47) otime 1609554203.226158833 (2021-01-01 21:23:23) item 23 key (147597 INODE_REF 146020) itemoff 13192 itemsize 27 index 508 namelen 17 name: kdeconnect-cli.mo item 24 key (147597 XATTR_ITEM 3817753667) itemoff 13116 itemsize 76 location key (0 UNKNOWN.0 0) type XATTR transid 757286 data_len 30 name_len 16 name: security.selinux data system_u:object_r:locale_t:s0 item 25 key (147597 EXTENT_DATA 0) itemoff 13063 itemsize 53 generation 757286 type 1 (regular) extent data disk byte 5798117376 nr 8192 extent data offset 0 nr 8192 ram 8192 extent compression 0 (none) item 26 key (147598 INODE_ITEM 0) itemoff 12903 itemsize 160 generation 757286 transid 757286 size 3122 nbytes 4096 block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x0(none) atime 1609554203.227159091 (2021-01-01 21:23:23) ctime 1609554203.227159091 (2021-01-01 21:23:23) mtime 1601653667.0 (2020-10-02 11:47:47) otime 1609554203.227159091 (2021-01-01 21:23:23) item 27 key (147598 INODE_REF 146020) itemoff 12875 itemsize 28 index 510 namelen 18 name: kdeconnect-core.mo item 28 key (147598 XATTR_ITEM 3817753667) itemoff 12799 itemsize 76 location key (0 UNKNOWN.0 0) type XATTR transid 757286 data_len 30 name_len 16 name: security.selinux data system_u:object_r:locale_t:s0 item 29 key (147598 EXTENT_DATA 0) itemoff 12746 itemsize 53 generation 757286 type 1 (regular) extent data disk byte 5798125568 nr 4096 extent data offset 0 nr 4096 ram 4096 extent compression 0 (none) item 30 key (147599 INODE_ITEM 0) itemoff 12586 itemsize 160 generation 757286 transid 757286 size 608 nbytes 608 block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x0(none) atime 1609554203.227159091 (2021-01-01 21:23:23) ctime 1609554203.227159091 (2021-01-01 21:23:23) mtime 1601653667.0 (2020-10-02 11:47:47) otime 1609554203.227159091 (2021-01-01 21:23:23) item 31 key (147599 INODE_REF 146020) itemoff 12548 itemsize 38 index 512 namelen 28 name: kdeconnect-fileitemaction.mo item 32 key (147599 XATTR_ITEM 3817753667) itemoff 12472 itemsize 76 location key (0 UNKNOWN.0 0) type XATTR transid 757286 data_len 30 name_len 16 name: security.selinux data system_u:object_r:locale_t:s0 item 33 key (147599 EXTENT_DATA 0) itemoff 11843 itemsize 629 generation 757286 type 0 (inline) inline extent data size 608 ram_bytes 608 compression 0 (none) item 34 key (147600 INODE_ITEM 0) itemoff 11683 itemsize 160 generation 757286 transid 757286 size 2243 nbytes 4096 block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x0(none) atime 1609554203.227159091 (2021-01-01 21:23:23) ctime 1609554203.227159091 (2021-01-01 21:23:23) mtime 1601653667.0 (2020-10-02 11:47:47) otime 1609554203.227159091 (2021-01-01 21:23:23) item 35 key (147600 INODE_REF 146020) itemoff 11650 itemsize 33 index 514 namelen 23 name: kdeconnect-indicator.mo item 36 key (147600 XATTR_ITEM 3817753667) itemoff 11574 itemsize 76 location key (0 UNKNOWN.0 0) type XATTR transid 757286 data_len 30 name_len 16 name: security.selinux data system_u:object_r:locale_t:s0 item 37 key (147600 EXTENT_DATA 0) itemoff 11521 itemsize 53 generation 757286 type 1 (regular) extent data disk byte 5798129664 nr 4096 extent data offset 0 nr 4096 ram 4096 extent compression 0 (none) item 38 key (147601 INODE_ITEM 0) itemoff 11361 itemsize 160 generation 757286 transid 757286 size 668 nbytes 668 block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x0(none) atime 1609554203.227159091 (2021-01-01 21:23:23) ctime 1609554203.227159091 (2021-01-01 21:23:23) mtime 1601653667.0 (2020-10-02 11:47:47) otime 1609554203.227159091 (2021-01-01 21:23:23) item 39 key (147601 INODE_REF 146020) itemoff 11327 itemsize 34 index 516 namelen 24 name: kdeconnect-interfaces.mo item 40 key (147601 XATTR_ITEM 3817753667) itemoff 11251 itemsize 76 location key (0 UNKNOWN.0 0) type XATTR transid 757286 data_len 30 name_len 16 name: security.selinux data system_u:object_r:locale_t:s0 item 41 key (147601 EXTENT_DATA 0) itemoff 10562 itemsize 689 generation 757286 type 0 (inline) inline extent data size 668 ram_bytes 668 compression 0 (none) item 42 key (147602 INODE_ITEM 0) itemoff 10402 itemsize 160 generation 757286 transid 757286 size 3276 nbytes 4096 block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x0(none) atime 1609554203.227159091 (2021-01-01 21:23:23) ctime 1609554203.227159091 (2021-01-01 21:23:23) mtime 1601653667.0 (2020-10-02 11:47:47) otime 1609554203.227159091 (2021-01-01 21:23:23) item 43 key (147602 INODE_REF 146020) itemoff 10375 itemsize 27 index 518 namelen 17 name: kdeconnect-kcm.mo item 44 key (147602 XATTR_ITEM 3817753667) itemoff 10299 itemsize 76 location key (0 UNKNOWN.0 0) type XATTR transid 757286 data_len 30 name_len 16 name: security.selinux data system_u:object_r:locale_t:s0 item 45 key (147602 EXTENT_DATA 0) itemoff 10246 itemsize 53 generation 757286 type 1 (regular) extent data disk byte 5798133760 nr 4096 extent data offset 0 nr 4096 ram 4096 extent compression 0 (none) item 46 key (147603 INODE_ITEM 0) itemoff 10086 itemsize 160 generation 757286 transid 757286 size 959 nbytes 959 block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x0(none) atime 1609554203.227159091 (2021-01-01 21:23:23) ctime 1609554203.227159091 (2021-01-01 21:23:23) mtime 1601653667.0 (2020-10-02 11:47:47) otime 1609554203.227159091 (2021-01-01 21:23:23) item 47 key (147603 INODE_REF 146020) itemoff 10058 itemsize 28 index 520 namelen 18 name: kdeconnect-kded.mo item 48 key (147603 XATTR_ITEM 3817753667) itemoff 9982 itemsize 76 location key (0 UNKNOWN.0 0) type XATTR transid 757286 data_len 30 name_len 16 name: security.selinux data system_u:object_r:locale_t:s0 item 49 key (147603 EXTENT_DATA 0) itemoff 9002 itemsize 980 generation 757286 type 0 (inline) inline extent data size 959 ram_bytes 959 compression 0 (none) item 50 key (147604 INODE_ITEM 0) itemoff 8842 itemsize 160 generation 757286 transid 757286 size 1033 nbytes 1033 block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x0(none) atime 1609554203.227159091 (2021-01-01 21:23:23) ctime 1609554203.227159091 (2021-01-01 21:23:23) mtime 1601653667.0 (2020-10-02 11:47:47) otime 1609554203.227159091 (2021-01-01 21:23:23) item 51 key (147604 INODE_REF 146020) itemoff 8815 itemsize 27 index 522 namelen 17 name: kdeconnect-kio.mo item 52 key (147604 XATTR_ITEM 3817753667) itemoff 8739 itemsize 76 location key (0 UNKNOWN.0 0) type XATTR transid 757286 data_len 30 name_len 16 name: security.selinux data system_u:object_r:locale_t:s0 ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-03-09 1:12 ` Neal Gompa @ 2021-03-09 19:04 ` Josef Bacik 2021-03-09 21:06 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-03-09 19:04 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 3/8/21 8:12 PM, Neal Gompa wrote: > On Mon, Mar 8, 2021 at 5:04 PM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 3/8/21 3:01 PM, Neal Gompa wrote: >>> On Mon, Mar 8, 2021 at 1:38 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>> >>>> On 3/5/21 8:03 PM, Neal Gompa wrote: >>>>> On Fri, Mar 5, 2021 at 5:01 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>> >>>>>> On 3/5/21 9:41 AM, Neal Gompa wrote: >>>>>>> On Fri, Mar 5, 2021 at 9:12 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>> >>>>>>>> On 3/4/21 6:54 PM, Neal Gompa wrote: >>>>>>>>> On Thu, Mar 4, 2021 at 3:25 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>> >>>>>>>>>> On 3/3/21 2:38 PM, Neal Gompa wrote: >>>>>>>>>>> On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 2/24/21 10:47 PM, Neal Gompa wrote: >>>>>>>>>>>>> On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2/24/21 9:23 AM, Neal Gompa wrote: >>>>>>>>>>>>>>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2/22/21 11:03 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> journal: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can anyone help? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you try >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other magic. Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Josef >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> No dice... >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 >>>>>>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> build, and then run >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr >>>>>>>>>>>>>>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind >>>>>>>>>>>>>>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 >>>>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>>>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 >>>>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> It's better, but still no dice... :( >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. >>>>>>>>>>>>>>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Nope, I see this in the journal: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Ok git pull for-neal, rebuild, then run >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances >>>>>>>>>>>>>>>>>>>>>> that the level doesn't match??". Thanks, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Tried rescue mount again after running that and got a stack trace in >>>>>>>>>>>>>>>>>>>>> the kernel, detailed in the following attached log. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with >>>>>>>>>>>>>>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a >>>>>>>>>>>>>>>>>>>> kernel with a fix for this >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> https://paste.centos.org/view/7b48aaea >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> and see if that gets you further. Thanks, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>>>>>>>>>>>>>>>>>> Killed >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The log from the journal is attached. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Ahh crud my bad, this should do it >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> https://paste.centos.org/view/ac2e61ef >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Patch doesn't apply (note it is patch 667 below): >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ah sorry, should have just sent you an iterative patch. You can take the above >>>>>>>>>>>>>>>> patch and just delete the hunk from volumes.c as you already have that applied >>>>>>>>>>>>>>>> and then it'll work. Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Failed with a weird error...? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt >>>>>>>>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Journal log with traceback attached. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Last one maybe? >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://paste.centos.org/view/80edd6fd >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Similar weird failure: >>>>>>>>>>>>> >>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>>>>>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. >>>>>>>>>>>>> >>>>>>>>>>>>> No crash in the journal this time, though: >>>>>>>>>>>>> >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Sorry Neal, you replied when I was in the middle of something and promptly >>>>>>>>>>>> forgot about it. I figured the fs root was fine, can you do the following so I >>>>>>>>>>>> can figure out from the error messages what might be wrong >>>>>>>>>>>> >>>>>>>>>>>> btrfs check --readonly >>>>>>>>>>>> btrfs restore -D >>>>>>>>>>>> btrfs restore -l >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> It didn't work.. Here's the output: >>>>>>>>>>> >>>>>>>>>>> [root@fedora ~]# btrfs check --readonly /dev/sdb3 >>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>> [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt >>>>>>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>>>>>> Couldn't setup device tree >>>>>>>>>>> Could not open root, trying backup super >>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>>>> Ignoring transid failure >>>>>>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>>>>>> Couldn't setup device tree >>>>>>>>>>> Could not open root, trying backup super >>>>>>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 >>>>>>>>>>> Could not open root, trying backup super >>>>>>>>>>> [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt >>>>>>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>>>>>> Couldn't setup device tree >>>>>>>>>>> Could not open root, trying backup super >>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>>>> Ignoring transid failure >>>>>>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>>>>>> Couldn't setup device tree >>>>>>>>>>> Could not open root, trying backup super >>>>>>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 >>>>>>>>>>> Could not open root, trying backup super >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hmm OK I think we want the neal magic for this one too, but before we go doing >>>>>>>>>> that can I get a >>>>>>>>>> >>>>>>>>>> btrfs inspect-internal -f /dev/whatever >>>>>>>>>> >>>>>>>>>> so I can make sure I'm not just blindly clobbering something. Thanks, >>>>>>>>>> >>>>>>>>> >>>>>>>>> Doesn't work, did you mean some other command? >>>>>>>>> >>>>>>>>> [root@fedora ~]# btrfs inspect-internal -f /dev/sdb3 >>>>>>>>> btrfs inspect-internal: unknown token '-f' >>>>>>>> >>>>>>>> Sigh, sorry, btrfs inspect-internal dump-super -f /dev/sdb3 >>>>>>>> >>>>>>> >>>>>> >>>>>> Ok I've pushed to the for-neal branch in my btrfs-progs, can you pull and make >>>>>> and then run >>>>>> >>>>>> ./btrfs-print-block /dev/sdb3 791281664 >>>>>> >>>>>> and capture everything it prints out? Thanks, >>>>>> >>>>> >>>>> Here's the output from the command. >>>>> >>>>> >>>> >>>> Hmm looks like the fs is offset a bit, can you do >>>> >>>> ./btrfs-print-block /dev/sdb3 799670272 >>>> >>> >>> This command caused my session to crash, but I do have a log of what >>> was captured before it crashed and attached it. >>> >>>> also while we're here can I get >>>> >>>> btrfs-find-root /dev/sdb3 >>>> >>> >>> This ran successfully and I've attached the output. >>> >> >> Ok we're going to try this again, and if it doesn't work it looks like your >> chunk root is ok, so I'll rig something up to make the translation work right, >> but for now lets do >> >> ./btrfs-print-block /dev/sdb3 792395776 >> > > I've attached the output from that command, which did run successfully. > Definitely need the translation, I pushed a new patch to the btrfs-progs branch for-neal. Pull, rebuild, and then run the same command again, hopefully this gives me what I want. Thanks, Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-03-09 19:04 ` Josef Bacik @ 2021-03-09 21:06 ` Neal Gompa 2021-03-09 21:56 ` Josef Bacik 0 siblings, 1 reply; 41+ messages in thread From: Neal Gompa @ 2021-03-09 21:06 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS [-- Attachment #1: Type: text/plain, Size: 17908 bytes --] On Tue, Mar 9, 2021 at 2:04 PM Josef Bacik <josef@toxicpanda.com> wrote: > > On 3/8/21 8:12 PM, Neal Gompa wrote: > > On Mon, Mar 8, 2021 at 5:04 PM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 3/8/21 3:01 PM, Neal Gompa wrote: > >>> On Mon, Mar 8, 2021 at 1:38 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>> > >>>> On 3/5/21 8:03 PM, Neal Gompa wrote: > >>>>> On Fri, Mar 5, 2021 at 5:01 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>> > >>>>>> On 3/5/21 9:41 AM, Neal Gompa wrote: > >>>>>>> On Fri, Mar 5, 2021 at 9:12 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>> > >>>>>>>> On 3/4/21 6:54 PM, Neal Gompa wrote: > >>>>>>>>> On Thu, Mar 4, 2021 at 3:25 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>> > >>>>>>>>>> On 3/3/21 2:38 PM, Neal Gompa wrote: > >>>>>>>>>>> On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 2/24/21 10:47 PM, Neal Gompa wrote: > >>>>>>>>>>>>> On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 2/24/21 9:23 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 2/22/21 11:03 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> journal: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can anyone help? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you try > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other magic. Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Josef > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> No dice... > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> build, and then run > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr > >>>>>>>>>>>>>>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind > >>>>>>>>>>>>>>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 > >>>>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>>>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 > >>>>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> It's better, but still no dice... :( > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. > >>>>>>>>>>>>>>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Nope, I see this in the journal: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options > >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums > >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots > >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time > >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Ok git pull for-neal, rebuild, then run > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances > >>>>>>>>>>>>>>>>>>>>>> that the level doesn't match??". Thanks, > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Tried rescue mount again after running that and got a stack trace in > >>>>>>>>>>>>>>>>>>>>> the kernel, detailed in the following attached log. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with > >>>>>>>>>>>>>>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a > >>>>>>>>>>>>>>>>>>>> kernel with a fix for this > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> https://paste.centos.org/view/7b48aaea > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> and see if that gets you further. Thanks, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>>>>>>>>>>>>>>>> Killed > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> The log from the journal is attached. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Ahh crud my bad, this should do it > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> https://paste.centos.org/view/ac2e61ef > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Patch doesn't apply (note it is patch 667 below): > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Ah sorry, should have just sent you an iterative patch. You can take the above > >>>>>>>>>>>>>>>> patch and just delete the hunk from volumes.c as you already have that applied > >>>>>>>>>>>>>>>> and then it'll work. Thanks, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Failed with a weird error...? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt > >>>>>>>>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Journal log with traceback attached. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Last one maybe? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> https://paste.centos.org/view/80edd6fd > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Similar weird failure: > >>>>>>>>>>>>> > >>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>>>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>>>>>>>>>>>> > >>>>>>>>>>>>> No crash in the journal this time, though: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options > >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums > >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots > >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time > >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled > >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents > >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 > >>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Sorry Neal, you replied when I was in the middle of something and promptly > >>>>>>>>>>>> forgot about it. I figured the fs root was fine, can you do the following so I > >>>>>>>>>>>> can figure out from the error messages what might be wrong > >>>>>>>>>>>> > >>>>>>>>>>>> btrfs check --readonly > >>>>>>>>>>>> btrfs restore -D > >>>>>>>>>>>> btrfs restore -l > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> It didn't work.. Here's the output: > >>>>>>>>>>> > >>>>>>>>>>> [root@fedora ~]# btrfs check --readonly /dev/sdb3 > >>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>> [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt > >>>>>>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>>>>>> Couldn't setup device tree > >>>>>>>>>>> Could not open root, trying backup super > >>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>>>> Ignoring transid failure > >>>>>>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>>>>>> Couldn't setup device tree > >>>>>>>>>>> Could not open root, trying backup super > >>>>>>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > >>>>>>>>>>> Could not open root, trying backup super > >>>>>>>>>>> [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt > >>>>>>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>>>>>> Couldn't setup device tree > >>>>>>>>>>> Could not open root, trying backup super > >>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>>>> Ignoring transid failure > >>>>>>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>>>>>> Couldn't setup device tree > >>>>>>>>>>> Could not open root, trying backup super > >>>>>>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > >>>>>>>>>>> Could not open root, trying backup super > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Hmm OK I think we want the neal magic for this one too, but before we go doing > >>>>>>>>>> that can I get a > >>>>>>>>>> > >>>>>>>>>> btrfs inspect-internal -f /dev/whatever > >>>>>>>>>> > >>>>>>>>>> so I can make sure I'm not just blindly clobbering something. Thanks, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> Doesn't work, did you mean some other command? > >>>>>>>>> > >>>>>>>>> [root@fedora ~]# btrfs inspect-internal -f /dev/sdb3 > >>>>>>>>> btrfs inspect-internal: unknown token '-f' > >>>>>>>> > >>>>>>>> Sigh, sorry, btrfs inspect-internal dump-super -f /dev/sdb3 > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> Ok I've pushed to the for-neal branch in my btrfs-progs, can you pull and make > >>>>>> and then run > >>>>>> > >>>>>> ./btrfs-print-block /dev/sdb3 791281664 > >>>>>> > >>>>>> and capture everything it prints out? Thanks, > >>>>>> > >>>>> > >>>>> Here's the output from the command. > >>>>> > >>>>> > >>>> > >>>> Hmm looks like the fs is offset a bit, can you do > >>>> > >>>> ./btrfs-print-block /dev/sdb3 799670272 > >>>> > >>> > >>> This command caused my session to crash, but I do have a log of what > >>> was captured before it crashed and attached it. > >>> > >>>> also while we're here can I get > >>>> > >>>> btrfs-find-root /dev/sdb3 > >>>> > >>> > >>> This ran successfully and I've attached the output. > >>> > >> > >> Ok we're going to try this again, and if it doesn't work it looks like your > >> chunk root is ok, so I'll rig something up to make the translation work right, > >> but for now lets do > >> > >> ./btrfs-print-block /dev/sdb3 792395776 > >> > > > > I've attached the output from that command, which did run successfully. > > > > Definitely need the translation, I pushed a new patch to the btrfs-progs branch > for-neal. Pull, rebuild, and then run the same command again, hopefully this > gives me what I want. Thanks, > Done and attached the output. -- 真実はいつも一つ！/ Always, there's only one truth! [-- Attachment #2: output-print-block4.log --] [-- Type: text/x-log, Size: 96487 bytes --] WARNING: could not setup extent tree, skipping it Couldn't setup device tree node 792395776 level 1 items 3 free space 490 generation 888895 owner ROOT_TREE node 792395776 flags 0x1(WRITTEN) backref revision 1 fs uuid f993ffa4-8801-4d57-a087-1c35fd6ece00 chunk uuid 7eff154b-3550-427e-98cb-7300b3d69ab3 key (EXTENT_TREE ROOT_ITEM 0) block 792428544 gen 888895 key (306 INODE_ITEM 0) block 799916032 gen 888895 key (378 INODE_ITEM 0) block 795443200 gen 888895 leaf 792428544 items 100 free space 1670 generation 888895 owner ROOT_TREE leaf 792428544 flags 0x1(WRITTEN) backref revision 1 fs uuid f993ffa4-8801-4d57-a087-1c35fd6ece00 chunk uuid 7eff154b-3550-427e-98cb-7300b3d69ab3 item 0 key (EXTENT_TREE ROOT_ITEM 0) itemoff 15844 itemsize 439 generation 888895 root_dirid 0 bytenr 791281664 level 2 refs 1 lastsnap 0 byte_limit 0 bytes_used 99532800 flags 0x0(none) uuid 00000000-0000-0000-0000-000000000000 drop key (0 UNKNOWN.0 0) level 0 item 1 key (DEV_TREE ROOT_ITEM 0) itemoff 15405 itemsize 439 generation 888539 root_dirid 0 bytenr 290979840 level 0 refs 1 lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none) uuid 00000000-0000-0000-0000-000000000000 drop key (0 UNKNOWN.0 0) level 0 item 2 key (FS_TREE INODE_REF 6) itemoff 15388 itemsize 17 index 0 namelen 7 name: default item 3 key (FS_TREE ROOT_ITEM 0) itemoff 14949 itemsize 439 generation 757276 root_dirid 256 bytenr 48431104 level 0 refs 1 lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none) uuid 6483c88d-dd94-4642-9d4f-b3b62d47cb37 ctransid 757276 otransid 0 stransid 0 rtransid 0 ctime 1609553954.398062897 (2021-01-01 21:19:14) otime 1584466114.0 (2020-03-17 13:28:34) drop key (0 UNKNOWN.0 0) level 0 item 4 key (FS_TREE ROOT_REF 258) itemoff 14927 itemsize 22 root ref key dirid 256 sequence 3 name home item 5 key (FS_TREE ROOT_REF 401) itemoff 14903 itemsize 24 root ref key dirid 256 sequence 4 name root00 item 6 key (ROOT_TREE_DIR INODE_ITEM 0) itemoff 14743 itemsize 160 generation 3 transid 0 size 0 nbytes 16384 block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0 sequence 0 flags 0x0(none) atime 1584466114.0 (2020-03-17 13:28:34) ctime 1584466114.0 (2020-03-17 13:28:34) mtime 1584466114.0 (2020-03-17 13:28:34) otime 1584466114.0 (2020-03-17 13:28:34) item 7 key (ROOT_TREE_DIR INODE_REF 6) itemoff 14731 itemsize 12 index 0 namelen 2 name: .. item 8 key (ROOT_TREE_DIR DIR_ITEM 2378154706) itemoff 14694 itemsize 37 location key (FS_TREE ROOT_ITEM 18446744073709551615) type DIR transid 0 data_len 0 name_len 7 name: default item 9 key (CSUM_TREE ROOT_ITEM 0) itemoff 14255 itemsize 439 generation 888895 root_dirid 0 bytenr 789741568 level 2 refs 1 lastsnap 0 byte_limit 0 bytes_used 139116544 flags 0x0(none) uuid 00000000-0000-0000-0000-000000000000 drop key (0 UNKNOWN.0 0) level 0 item 10 key (UUID_TREE ROOT_ITEM 0) itemoff 13816 itemsize 439 generation 777114 root_dirid 0 bytenr 72019689472 level 0 refs 1 lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none) uuid 00000000-0000-0000-0000-000000000000 drop key (0 UNKNOWN.0 0) level 0 item 11 key (257 INODE_ITEM 0) itemoff 13656 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 124776873984 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 475986 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.399257555 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 12 key (257 EXTENT_DATA 0) itemoff 13603 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 10157371392 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 13 key (258 ROOT_ITEM 0) itemoff 13164 itemsize 439 generation 888895 root_dirid 256 bytenr 790118400 level 2 refs 1 lastsnap 0 byte_limit 0 bytes_used 953942016 flags 0x0(none) uuid 65e282be-3624-814f-a161-f510a79e946a ctransid 888895 otransid 8 stransid 0 rtransid 0 ctime 1613320613.824154996 (2021-02-14 11:36:53) otime 1584466115.277451365 (2020-03-17 13:28:35) drop key (0 UNKNOWN.0 0) level 0 item 14 key (258 ROOT_BACKREF 5) itemoff 13142 itemsize 22 root backref key dirid 256 sequence 3 name home item 15 key (258 ROOT_REF 362) itemoff 13116 itemsize 26 root ref key dirid 4504931 sequence 3 name chromium item 16 key (259 INODE_ITEM 0) itemoff 12956 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 71211417600 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 271650 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.400257562 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 17 key (259 EXTENT_DATA 0) itemoff 12903 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 11649806336 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 18 key (260 INODE_ITEM 0) itemoff 12743 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 61304209408 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 233857 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.401257568 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 19 key (260 EXTENT_DATA 0) itemoff 12690 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 13132550144 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 20 key (261 INODE_ITEM 0) itemoff 12530 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 60248817664 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 229831 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.402257575 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 21 key (261 EXTENT_DATA 0) itemoff 12477 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 13133074432 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 22 key (262 INODE_ITEM 0) itemoff 12317 itemsize 160 generation 888491 transid 888491 size 262144 nbytes 29348069376 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 111954 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613247298.278699122 (2021-02-13 15:14:58) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 23 key (262 EXTENT_DATA 0) itemoff 12264 itemsize 53 generation 888491 type 1 (regular) extent data disk byte 45669081088 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 24 key (263 INODE_ITEM 0) itemoff 12104 itemsize 160 generation 888191 transid 888191 size 262144 nbytes 41881436160 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 159765 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613237107.753888991 (2021-02-13 12:25:07) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 25 key (263 EXTENT_DATA 0) itemoff 12051 itemsize 53 generation 888191 type 1 (regular) extent data disk byte 21485936640 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 26 key (264 INODE_ITEM 0) itemoff 11891 itemsize 160 generation 888775 transid 888775 size 262144 nbytes 55252090880 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 210770 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613305178.73830080 (2021-02-14 07:19:38) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 27 key (264 EXTENT_DATA 0) itemoff 11838 itemsize 53 generation 888775 type 1 (regular) extent data disk byte 16720691200 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 28 key (265 INODE_ITEM 0) itemoff 11678 itemsize 160 generation 888570 transid 888570 size 262144 nbytes 52726857728 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 201137 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613274126.48693706 (2021-02-13 22:42:06) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 29 key (265 EXTENT_DATA 0) itemoff 11625 itemsize 53 generation 888570 type 1 (regular) extent data disk byte 22255992832 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 30 key (267 INODE_ITEM 0) itemoff 11465 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 46453489664 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 177206 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.406257601 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 31 key (267 EXTENT_DATA 0) itemoff 11412 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 14565695488 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 32 key (268 INODE_ITEM 0) itemoff 11252 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 44335890432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 169128 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.407257607 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 33 key (268 EXTENT_DATA 0) itemoff 11199 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 14566248448 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 34 key (269 INODE_ITEM 0) itemoff 11039 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 45505314816 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 173589 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.408257614 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 35 key (269 EXTENT_DATA 0) itemoff 10986 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 14963179520 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 36 key (270 INODE_ITEM 0) itemoff 10826 itemsize 160 generation 888775 transid 888775 size 262144 nbytes 35990274048 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 137292 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613305178.84830156 (2021-02-14 07:19:38) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 37 key (270 EXTENT_DATA 0) itemoff 10773 itemsize 53 generation 888775 type 1 (regular) extent data disk byte 21683920896 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 38 key (271 INODE_ITEM 0) itemoff 10613 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 21695823872 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 82763 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.409257620 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 39 key (271 EXTENT_DATA 0) itemoff 10560 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 15340331008 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 40 key (272 INODE_ITEM 0) itemoff 10400 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 30135287808 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 114957 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.403257581 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 41 key (272 EXTENT_DATA 0) itemoff 10347 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 13134123008 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 42 key (273 INODE_ITEM 0) itemoff 10187 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 36564893696 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 139484 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.409257620 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 43 key (273 EXTENT_DATA 0) itemoff 10134 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 15340593152 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 44 key (274 INODE_ITEM 0) itemoff 9974 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 29774839808 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 113582 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.418257679 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 45 key (274 EXTENT_DATA 0) itemoff 9921 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 15343153152 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 46 key (275 INODE_ITEM 0) itemoff 9761 itemsize 160 generation 888497 transid 888497 size 262144 nbytes 16281239552 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 62108 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613247817.898916220 (2021-02-13 15:23:37) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 47 key (275 EXTENT_DATA 0) itemoff 9708 itemsize 53 generation 888497 type 1 (regular) extent data disk byte 45806456832 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 48 key (276 INODE_ITEM 0) itemoff 9548 itemsize 160 generation 888609 transid 888609 size 262144 nbytes 21069299712 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 80373 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613278893.732881392 (2021-02-14 00:01:33) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 49 key (276 EXTENT_DATA 0) itemoff 9495 itemsize 53 generation 888609 type 1 (regular) extent data disk byte 21842087936 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 50 key (277 INODE_ITEM 0) itemoff 9335 itemsize 160 generation 888745 transid 888745 size 262144 nbytes 30340808704 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 115741 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613301802.958307217 (2021-02-14 06:23:22) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 51 key (277 EXTENT_DATA 0) itemoff 9282 itemsize 53 generation 888745 type 1 (regular) extent data disk byte 14565433344 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 52 key (278 INODE_ITEM 0) itemoff 9122 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 34646786048 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 132167 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.424257718 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 53 key (278 EXTENT_DATA 0) itemoff 9069 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 23840440320 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 54 key (280 INODE_ITEM 0) itemoff 8909 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 28001435648 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 106817 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.423257711 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 55 key (280 EXTENT_DATA 0) itemoff 8856 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 21482881024 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 56 key (281 INODE_ITEM 0) itemoff 8696 itemsize 160 generation 888609 transid 888609 size 262144 nbytes 25640042496 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 97809 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613278893.738881430 (2021-02-14 00:01:33) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 57 key (281 EXTENT_DATA 0) itemoff 8643 itemsize 53 generation 888609 type 1 (regular) extent data disk byte 34052096000 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 58 key (282 INODE_ITEM 0) itemoff 8483 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 18623234048 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 71042 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.426257731 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 59 key (282 EXTENT_DATA 0) itemoff 8430 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 24466911232 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 60 key (283 INODE_ITEM 0) itemoff 8270 itemsize 160 generation 888609 transid 888609 size 262144 nbytes 23184801792 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 88443 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613278893.739881436 (2021-02-14 00:01:33) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 61 key (283 EXTENT_DATA 0) itemoff 8217 itemsize 53 generation 888609 type 1 (regular) extent data disk byte 34052620288 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 62 key (284 INODE_ITEM 0) itemoff 8057 itemsize 160 generation 888609 transid 888609 size 262144 nbytes 20114046976 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 76729 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613278893.739881436 (2021-02-14 00:01:33) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 63 key (284 EXTENT_DATA 0) itemoff 8004 itemsize 53 generation 888609 type 1 (regular) extent data disk byte 34052882432 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 64 key (285 INODE_ITEM 0) itemoff 7844 itemsize 160 generation 888785 transid 888785 size 262144 nbytes 16836984832 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 64228 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613305414.936411978 (2021-02-14 07:23:34) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 65 key (285 EXTENT_DATA 0) itemoff 7791 itemsize 53 generation 888785 type 1 (regular) extent data disk byte 14563766272 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 66 key (286 INODE_ITEM 0) itemoff 7631 itemsize 160 generation 888497 transid 888497 size 262144 nbytes 7678984192 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 29293 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613247817.900916228 (2021-02-13 15:23:37) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 67 key (286 EXTENT_DATA 0) itemoff 7578 itemsize 53 generation 888497 type 1 (regular) extent data disk byte 45853130752 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 68 key (287 INODE_ITEM 0) itemoff 7418 itemsize 160 generation 888401 transid 888401 size 262144 nbytes 6104023040 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 23285 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613237126.565919320 (2021-02-13 12:25:26) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 69 key (287 EXTENT_DATA 0) itemoff 7365 itemsize 53 generation 888401 type 1 (regular) extent data disk byte 34575073280 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 70 key (288 INODE_ITEM 0) itemoff 7205 itemsize 160 generation 888570 transid 888570 size 262144 nbytes 18642108416 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 71114 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613274126.51693725 (2021-02-13 22:42:06) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 71 key (288 EXTENT_DATA 0) itemoff 7152 itemsize 53 generation 888570 type 1 (regular) extent data disk byte 35192918016 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 72 key (289 INODE_ITEM 0) itemoff 6992 itemsize 160 generation 888411 transid 888411 size 262144 nbytes 15332540416 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 58489 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613237140.214941329 (2021-02-13 12:25:40) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 73 key (289 EXTENT_DATA 0) itemoff 6939 itemsize 53 generation 888411 type 1 (regular) extent data disk byte 35187408896 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 74 key (290 INODE_ITEM 0) itemoff 6779 itemsize 160 generation 888667 transid 888667 size 262144 nbytes 17579376640 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 67060 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613288488.930380755 (2021-02-14 02:41:28) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 75 key (290 EXTENT_DATA 0) itemoff 6726 itemsize 53 generation 888667 type 1 (regular) extent data disk byte 14963441664 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 76 key (291 INODE_ITEM 0) itemoff 6566 itemsize 160 generation 888894 transid 888894 size 262144 nbytes 18659934208 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 71182 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320603.95027840 (2021-02-14 11:36:43) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 77 key (291 EXTENT_DATA 0) itemoff 6513 itemsize 53 generation 888894 type 1 (regular) extent data disk byte 23769411584 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 78 key (292 INODE_ITEM 0) itemoff 6353 itemsize 160 generation 888894 transid 888894 size 262144 nbytes 26251886592 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 100143 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320603.80027742 (2021-02-14 11:36:43) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 79 key (292 EXTENT_DATA 0) itemoff 6300 itemsize 53 generation 888894 type 1 (regular) extent data disk byte 14565171200 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 80 key (293 INODE_ITEM 0) itemoff 6140 itemsize 160 generation 888785 transid 888785 size 262144 nbytes 20565196800 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 78450 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613305414.949412065 (2021-02-14 07:23:34) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 81 key (293 EXTENT_DATA 0) itemoff 6087 itemsize 53 generation 888785 type 1 (regular) extent data disk byte 15340855296 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 82 key (294 INODE_ITEM 0) itemoff 5927 itemsize 160 generation 888785 transid 888785 size 262144 nbytes 9726328832 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 37103 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613305414.953412092 (2021-02-14 07:23:34) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 83 key (294 EXTENT_DATA 0) itemoff 5874 itemsize 53 generation 888785 type 1 (regular) extent data disk byte 15445594112 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 84 key (295 INODE_ITEM 0) itemoff 5714 itemsize 160 generation 888894 transid 888894 size 262144 nbytes 78829846528 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 300712 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320603.81027749 (2021-02-14 11:36:43) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 85 key (295 EXTENT_DATA 0) itemoff 5661 itemsize 53 generation 888894 type 1 (regular) extent data disk byte 14566510592 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 86 key (297 INODE_ITEM 0) itemoff 5501 itemsize 160 generation 873604 transid 873604 size 262144 nbytes 2826436608 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 10782 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1612887527.220817093 (2021-02-09 11:18:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 87 key (297 EXTENT_DATA 0) itemoff 5448 itemsize 53 generation 873604 type 1 (regular) extent data disk byte 14992502784 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 88 key (298 INODE_ITEM 0) itemoff 5288 itemsize 160 generation 888023 transid 888023 size 262144 nbytes 4350017536 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 16594 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613236483.770926392 (2021-02-13 12:14:43) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 89 key (298 EXTENT_DATA 0) itemoff 5235 itemsize 53 generation 888023 type 1 (regular) extent data disk byte 45541822464 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 90 key (299 INODE_ITEM 0) itemoff 5075 itemsize 160 generation 886667 transid 886667 size 262144 nbytes 9928966144 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 37876 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613161668.950049490 (2021-02-12 15:27:48) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 91 key (299 EXTENT_DATA 0) itemoff 5022 itemsize 53 generation 886667 type 1 (regular) extent data disk byte 43416514560 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 92 key (300 INODE_ITEM 0) itemoff 4862 itemsize 160 generation 888800 transid 888800 size 262144 nbytes 28255977472 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 107788 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613305437.393561914 (2021-02-14 07:23:57) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 93 key (300 EXTENT_DATA 0) itemoff 4809 itemsize 53 generation 888800 type 1 (regular) extent data disk byte 13133336576 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 94 key (301 INODE_ITEM 0) itemoff 4649 itemsize 160 generation 888872 transid 888872 size 262144 nbytes 8302362624 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 31671 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613317279.724811720 (2021-02-14 10:41:19) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 95 key (301 EXTENT_DATA 0) itemoff 4596 itemsize 53 generation 888872 type 1 (regular) extent data disk byte 15447052288 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 96 key (302 INODE_ITEM 0) itemoff 4436 itemsize 160 generation 888872 transid 888872 size 262144 nbytes 5391515648 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 20567 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613317279.727811739 (2021-02-14 10:41:19) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 97 key (302 EXTENT_DATA 0) itemoff 4383 itemsize 53 generation 888872 type 1 (regular) extent data disk byte 15447314432 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 98 key (303 INODE_ITEM 0) itemoff 4223 itemsize 160 generation 888872 transid 888872 size 262144 nbytes 8093171712 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 30873 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613317279.730811759 (2021-02-14 10:41:19) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 99 key (303 EXTENT_DATA 0) itemoff 4170 itemsize 53 generation 888872 type 1 (regular) extent data disk byte 15778799616 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) leaf 799916032 items 111 free space 1381 generation 888895 owner ROOT_TREE leaf 799916032 flags 0x1(WRITTEN) backref revision 1 fs uuid f993ffa4-8801-4d57-a087-1c35fd6ece00 chunk uuid 7eff154b-3550-427e-98cb-7300b3d69ab3 item 0 key (306 INODE_ITEM 0) itemoff 16123 itemsize 160 generation 888872 transid 888872 size 262144 nbytes 2963537920 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 11305 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613317279.733811779 (2021-02-14 10:41:19) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 1 key (306 EXTENT_DATA 0) itemoff 16070 itemsize 53 generation 888872 type 1 (regular) extent data disk byte 15779848192 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 2 key (307 INODE_ITEM 0) itemoff 15910 itemsize 160 generation 885295 transid 885295 size 262144 nbytes 16252928 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 62 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613083281.309313744 (2021-02-11 17:41:21) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 3 key (307 EXTENT_DATA 0) itemoff 15857 itemsize 53 generation 885295 type 1 (regular) extent data disk byte 56227926016 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 4 key (308 INODE_ITEM 0) itemoff 15697 itemsize 160 generation 888667 transid 888667 size 262144 nbytes 4135059456 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 15774 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613288488.951380918 (2021-02-14 02:41:28) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 5 key (308 EXTENT_DATA 0) itemoff 15644 itemsize 53 generation 888667 type 1 (regular) extent data disk byte 15779061760 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 6 key (309 INODE_ITEM 0) itemoff 15484 itemsize 160 generation 888872 transid 888872 size 262144 nbytes 7623409664 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 29081 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613317279.737811805 (2021-02-14 10:41:19) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 7 key (309 EXTENT_DATA 0) itemoff 15431 itemsize 53 generation 888872 type 1 (regular) extent data disk byte 15780110336 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 8 key (310 INODE_ITEM 0) itemoff 15271 itemsize 160 generation 888791 transid 888791 size 262144 nbytes 2747793408 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 10482 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613305426.720490654 (2021-02-14 07:23:46) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 9 key (310 EXTENT_DATA 0) itemoff 15218 itemsize 53 generation 888791 type 1 (regular) extent data disk byte 15447576576 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 10 key (311 INODE_ITEM 0) itemoff 15058 itemsize 160 generation 888791 transid 888791 size 262144 nbytes 1986789376 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 7579 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613305426.721490660 (2021-02-14 07:23:46) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 11 key (311 EXTENT_DATA 0) itemoff 15005 itemsize 53 generation 888791 type 1 (regular) extent data disk byte 15778537472 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 12 key (312 INODE_ITEM 0) itemoff 14845 itemsize 160 generation 885296 transid 885296 size 262144 nbytes 402391040 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1535 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613083311.711487513 (2021-02-11 17:41:51) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 13 key (312 EXTENT_DATA 0) itemoff 14792 itemsize 53 generation 885296 type 1 (regular) extent data disk byte 7023857664 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 14 key (313 INODE_ITEM 0) itemoff 14632 itemsize 160 generation 888791 transid 888791 size 262144 nbytes 5081137152 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 19383 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613305426.722490667 (2021-02-14 07:23:46) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 15 key (313 EXTENT_DATA 0) itemoff 14579 itemsize 53 generation 888791 type 1 (regular) extent data disk byte 15779323904 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 16 key (314 INODE_ITEM 0) itemoff 14419 itemsize 160 generation 888572 transid 888572 size 262144 nbytes 1959788544 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 7476 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613274128.899711649 (2021-02-13 22:42:08) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 17 key (314 EXTENT_DATA 0) itemoff 14366 itemsize 53 generation 888572 type 1 (regular) extent data disk byte 14564909056 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 18 key (315 INODE_ITEM 0) itemoff 14206 itemsize 160 generation 888662 transid 888662 size 262144 nbytes 6597378048 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 25167 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613288456.832130372 (2021-02-14 02:40:56) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 19 key (315 EXTENT_DATA 0) itemoff 14153 itemsize 53 generation 888662 type 1 (regular) extent data disk byte 34052358144 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 20 key (316 INODE_ITEM 0) itemoff 13993 itemsize 160 generation 885555 transid 885555 size 262144 nbytes 3217555456 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 12274 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613116488.812155480 (2021-02-12 02:54:48) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 21 key (316 EXTENT_DATA 0) itemoff 13940 itemsize 53 generation 885555 type 1 (regular) extent data disk byte 14620393472 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 22 key (317 INODE_ITEM 0) itemoff 13780 itemsize 160 generation 888662 transid 888662 size 262144 nbytes 2383937536 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9094 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613288456.814130231 (2021-02-14 02:40:56) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 23 key (317 EXTENT_DATA 0) itemoff 13727 itemsize 53 generation 888662 type 1 (regular) extent data disk byte 15778275328 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 24 key (318 INODE_ITEM 0) itemoff 13567 itemsize 160 generation 887740 transid 887740 size 262144 nbytes 7526154240 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 28710 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613231212.250503685 (2021-02-13 10:46:52) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 25 key (318 EXTENT_DATA 0) itemoff 13514 itemsize 53 generation 887740 type 1 (regular) extent data disk byte 23769149440 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 26 key (319 INODE_ITEM 0) itemoff 13354 itemsize 160 generation 887741 transid 887741 size 262144 nbytes 3969646592 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 15143 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613231235.514520835 (2021-02-13 10:47:15) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 27 key (319 EXTENT_DATA 0) itemoff 13301 itemsize 53 generation 887741 type 1 (regular) extent data disk byte 15221653504 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 28 key (320 INODE_ITEM 0) itemoff 13141 itemsize 160 generation 888662 transid 888662 size 262144 nbytes 4154982400 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 15850 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613288456.815130239 (2021-02-14 02:40:56) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 29 key (320 EXTENT_DATA 0) itemoff 13088 itemsize 53 generation 888662 type 1 (regular) extent data disk byte 15780470784 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 30 key (321 INODE_ITEM 0) itemoff 12928 itemsize 160 generation 886055 transid 886055 size 262144 nbytes 7048003584 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 26886 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613137069.527177495 (2021-02-12 08:37:49) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 31 key (321 EXTENT_DATA 0) itemoff 12875 itemsize 53 generation 886055 type 1 (regular) extent data disk byte 23635169280 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 32 key (324 INODE_ITEM 0) itemoff 12715 itemsize 160 generation 885295 transid 885295 size 262144 nbytes 1190133760 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4540 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613083281.322313818 (2021-02-11 17:41:21) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 33 key (324 EXTENT_DATA 0) itemoff 12662 itemsize 53 generation 885295 type 1 (regular) extent data disk byte 56329633792 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 34 key (325 INODE_ITEM 0) itemoff 12502 itemsize 160 generation 888037 transid 888037 size 262144 nbytes 2895642624 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 11046 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613237024.456754688 (2021-02-13 12:23:44) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 35 key (325 EXTENT_DATA 0) itemoff 12449 itemsize 53 generation 888037 type 1 (regular) extent data disk byte 45852868608 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 36 key (327 INODE_ITEM 0) itemoff 12289 itemsize 160 generation 886682 transid 886682 size 262144 nbytes 738197504 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2816 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613161671.643051596 (2021-02-12 15:27:51) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 37 key (327 EXTENT_DATA 0) itemoff 12236 itemsize 53 generation 886682 type 1 (regular) extent data disk byte 43576602624 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 38 key (328 INODE_ITEM 0) itemoff 12076 itemsize 160 generation 765522 transid 765522 size 262144 nbytes 7282622464 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 27781 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874442.121905657 (2021-01-05 14:20:42) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 39 key (328 EXTENT_DATA 0) itemoff 12023 itemsize 53 generation 765522 type 1 (regular) extent data disk byte 2494529536 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 40 key (330 INODE_ITEM 0) itemoff 11863 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 39814430720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 151880 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.418257679 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 41 key (330 EXTENT_DATA 0) itemoff 11810 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 15343427584 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 42 key (331 INODE_ITEM 0) itemoff 11650 itemsize 160 generation 888895 transid 888895 size 262144 nbytes 29650845696 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 113109 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613320638.418257679 (2021-02-14 11:37:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 43 key (331 EXTENT_DATA 0) itemoff 11597 itemsize 53 generation 888895 type 1 (regular) extent data disk byte 15445856256 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 44 key (332 INODE_ITEM 0) itemoff 11437 itemsize 160 generation 888037 transid 888037 size 262144 nbytes 1934622720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 7380 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613237024.459754693 (2021-02-13 12:23:44) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 45 key (332 EXTENT_DATA 0) itemoff 11384 itemsize 53 generation 888037 type 1 (regular) extent data disk byte 45853655040 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 46 key (334 INODE_ITEM 0) itemoff 11224 itemsize 160 generation 885295 transid 885295 size 262144 nbytes 4255121408 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 16232 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613083281.326313841 (2021-02-11 17:41:21) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 47 key (334 EXTENT_DATA 0) itemoff 11171 itemsize 53 generation 885295 type 1 (regular) extent data disk byte 56379187200 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 48 key (335 INODE_ITEM 0) itemoff 11011 itemsize 160 generation 888662 transid 888662 size 262144 nbytes 1147404288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4377 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613288456.818130262 (2021-02-14 02:40:56) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 49 key (335 EXTENT_DATA 0) itemoff 10958 itemsize 53 generation 888662 type 1 (regular) extent data disk byte 21501317120 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 50 key (336 INODE_ITEM 0) itemoff 10798 itemsize 160 generation 887742 transid 887742 size 262144 nbytes 1202978816 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4589 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613231238.516523047 (2021-02-13 10:47:18) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 51 key (336 EXTENT_DATA 0) itemoff 10745 itemsize 53 generation 887742 type 1 (regular) extent data disk byte 20310740992 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 52 key (338 INODE_ITEM 0) itemoff 10585 itemsize 160 generation 888576 transid 888576 size 262144 nbytes 1610350592 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 6143 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613274237.390886762 (2021-02-13 22:43:57) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 53 key (338 EXTENT_DATA 0) itemoff 10532 itemsize 53 generation 888576 type 1 (regular) extent data disk byte 15446528000 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 54 key (339 INODE_ITEM 0) itemoff 10372 itemsize 160 generation 885295 transid 885295 size 262144 nbytes 1329332224 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5071 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613083281.331313869 (2021-02-11 17:41:21) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 55 key (339 EXTENT_DATA 0) itemoff 10319 itemsize 53 generation 885295 type 1 (regular) extent data disk byte 56422019072 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 56 key (340 INODE_ITEM 0) itemoff 10159 itemsize 160 generation 885295 transid 885295 size 262144 nbytes 623902720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2380 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613083281.333313881 (2021-02-11 17:41:21) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 57 key (340 EXTENT_DATA 0) itemoff 10106 itemsize 53 generation 885295 type 1 (regular) extent data disk byte 56492699648 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 58 key (341 INODE_ITEM 0) itemoff 9946 itemsize 160 generation 0 transid 765522 size 0 nbytes 149159936 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 569 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1608658116.993176122 (2020-12-22 12:28:36) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 59 key (342 INODE_ITEM 0) itemoff 9786 itemsize 160 generation 885295 transid 885295 size 262144 nbytes 1827930112 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 6973 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613083281.334313886 (2021-02-11 17:41:21) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 60 key (342 EXTENT_DATA 0) itemoff 9733 itemsize 53 generation 885295 type 1 (regular) extent data disk byte 56493694976 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 61 key (343 INODE_ITEM 0) itemoff 9573 itemsize 160 generation 888037 transid 888037 size 262144 nbytes 850132992 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3243 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613237024.461754696 (2021-02-13 12:23:44) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 62 key (343 EXTENT_DATA 0) itemoff 9520 itemsize 53 generation 888037 type 1 (regular) extent data disk byte 45853917184 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 63 key (344 INODE_ITEM 0) itemoff 9360 itemsize 160 generation 888488 transid 888488 size 262144 nbytes 1183580160 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4515 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613247217.330353725 (2021-02-13 15:13:37) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 64 key (344 EXTENT_DATA 0) itemoff 9307 itemsize 53 generation 888488 type 1 (regular) extent data disk byte 35192655872 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 65 key (346 INODE_ITEM 0) itemoff 9147 itemsize 160 generation 888573 transid 888573 size 262144 nbytes 330825728 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1262 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613274161.636051269 (2021-02-13 22:42:41) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 66 key (346 EXTENT_DATA 0) itemoff 9094 itemsize 53 generation 888573 type 1 (regular) extent data disk byte 34582462464 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 67 key (349 INODE_ITEM 0) itemoff 8934 itemsize 160 generation 885295 transid 885295 size 262144 nbytes 359923712 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1373 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613083281.337313904 (2021-02-11 17:41:21) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 68 key (349 EXTENT_DATA 0) itemoff 8881 itemsize 53 generation 885295 type 1 (regular) extent data disk byte 56537968640 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 69 key (350 INODE_ITEM 0) itemoff 8721 itemsize 160 generation 885295 transid 885295 size 262144 nbytes 395837440 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1510 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613083281.337313904 (2021-02-11 17:41:21) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 70 key (350 EXTENT_DATA 0) itemoff 8668 itemsize 53 generation 885295 type 1 (regular) extent data disk byte 56538255360 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 71 key (355 INODE_ITEM 0) itemoff 8508 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 205258752 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 783 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 72 key (355 EXTENT_DATA 0) itemoff 8455 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2406526976 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 73 key (356 INODE_ITEM 0) itemoff 8295 itemsize 160 generation 885295 transid 885295 size 262144 nbytes 1257242624 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4796 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613083281.339313915 (2021-02-11 17:41:21) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 74 key (356 EXTENT_DATA 0) itemoff 8242 itemsize 53 generation 885295 type 1 (regular) extent data disk byte 56538787840 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 75 key (358 INODE_ITEM 0) itemoff 8082 itemsize 160 generation 885295 transid 885295 size 262144 nbytes 604504064 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2306 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613083281.340313921 (2021-02-11 17:41:21) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 76 key (358 EXTENT_DATA 0) itemoff 8029 itemsize 53 generation 885295 type 1 (regular) extent data disk byte 56541077504 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 77 key (361 INODE_ITEM 0) itemoff 7869 itemsize 160 generation 881621 transid 881621 size 262144 nbytes 17825792 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 68 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613048351.684578533 (2021-02-11 07:59:11) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 78 key (361 EXTENT_DATA 0) itemoff 7816 itemsize 53 generation 881621 type 1 (regular) extent data disk byte 12058468352 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 79 key (362 ROOT_ITEM 0) itemoff 7377 itemsize 439 generation 757300 root_dirid 256 bytenr 38484148224 level 2 refs 1 lastsnap 0 byte_limit 0 bytes_used 694583296 flags 0x0(none) uuid ec844a3e-03cd-d043-bc80-933344d28167 ctransid 757300 otransid 693308 stransid 0 rtransid 0 ctime 1609554583.179666842 (2021-01-01 21:29:43) otime 1605904563.818142101 (2020-11-20 15:36:03) drop key (0 UNKNOWN.0 0) level 0 item 80 key (362 ROOT_BACKREF 258) itemoff 7351 itemsize 26 root backref key dirid 4504931 sequence 3 name chromium item 81 key (363 INODE_ITEM 0) itemoff 7191 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 8126464 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 31 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 82 key (363 EXTENT_DATA 0) itemoff 7138 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2424549376 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 83 key (364 INODE_ITEM 0) itemoff 6978 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 3670016 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 84 key (364 EXTENT_DATA 0) itemoff 6925 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2425049088 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 85 key (365 INODE_ITEM 0) itemoff 6765 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 2359296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 86 key (365 EXTENT_DATA 0) itemoff 6712 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2428366848 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 87 key (366 INODE_ITEM 0) itemoff 6552 itemsize 160 generation 693323 transid 693323 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1605905228.404708425 (2020-11-20 15:47:08) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 88 key (366 EXTENT_DATA 0) itemoff 6499 itemsize 53 generation 693323 type 1 (regular) extent data disk byte 3760914432 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 89 key (367 INODE_ITEM 0) itemoff 6339 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 2359296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 90 key (367 EXTENT_DATA 0) itemoff 6286 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2428628992 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 91 key (368 INODE_ITEM 0) itemoff 6126 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 3145728 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 92 key (368 EXTENT_DATA 0) itemoff 6073 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2429849600 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 93 key (369 INODE_ITEM 0) itemoff 5913 itemsize 160 generation 693327 transid 693327 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1605905364.602764643 (2020-11-20 15:49:24) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 94 key (369 EXTENT_DATA 0) itemoff 5860 itemsize 53 generation 693327 type 1 (regular) extent data disk byte 4248252416 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 95 key (370 INODE_ITEM 0) itemoff 5700 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 35127296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 134 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 96 key (370 EXTENT_DATA 0) itemoff 5647 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2430111744 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 97 key (371 INODE_ITEM 0) itemoff 5487 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 3407872 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 13 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 98 key (371 EXTENT_DATA 0) itemoff 5434 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2430373888 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 99 key (372 INODE_ITEM 0) itemoff 5274 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 1835008 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 7 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 100 key (372 EXTENT_DATA 0) itemoff 5221 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2434781184 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 101 key (373 INODE_ITEM 0) itemoff 5061 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 2883584 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 102 key (373 EXTENT_DATA 0) itemoff 5008 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2435043328 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 103 key (374 INODE_ITEM 0) itemoff 4848 itemsize 160 generation 802241 transid 802241 size 262144 nbytes 2883584 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1610592670.509339297 (2021-01-13 21:51:10) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 104 key (374 EXTENT_DATA 0) itemoff 4795 itemsize 53 generation 802241 type 1 (regular) extent data disk byte 11648925696 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 105 key (375 INODE_ITEM 0) itemoff 4635 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 30408704 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 116 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 106 key (375 EXTENT_DATA 0) itemoff 4582 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2442563584 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 107 key (376 INODE_ITEM 0) itemoff 4422 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 30408704 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 116 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 108 key (376 EXTENT_DATA 0) itemoff 4369 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2444275712 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 109 key (377 INODE_ITEM 0) itemoff 4209 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 2359296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 110 key (377 EXTENT_DATA 0) itemoff 4156 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2444840960 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) leaf 795443200 items 164 free space 1692 generation 888895 owner ROOT_TREE leaf 795443200 flags 0x1(WRITTEN) backref revision 1 fs uuid f993ffa4-8801-4d57-a087-1c35fd6ece00 chunk uuid 7eff154b-3550-427e-98cb-7300b3d69ab3 item 0 key (378 INODE_ITEM 0) itemoff 16123 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 1 key (378 EXTENT_DATA 0) itemoff 16070 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2445447168 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 2 key (379 INODE_ITEM 0) itemoff 15910 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 2097152 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 8 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 3 key (379 EXTENT_DATA 0) itemoff 15857 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2312896512 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 4 key (380 INODE_ITEM 0) itemoff 15697 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 1572864 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 6 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 5 key (380 EXTENT_DATA 0) itemoff 15644 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2313375744 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 6 key (381 INODE_ITEM 0) itemoff 15484 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 7 key (381 EXTENT_DATA 0) itemoff 15431 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2314407936 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 8 key (382 INODE_ITEM 0) itemoff 15271 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 56360960 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 215 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 9 key (382 EXTENT_DATA 0) itemoff 15218 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2314670080 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 10 key (383 INODE_ITEM 0) itemoff 15058 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 1572864 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 6 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 11 key (383 EXTENT_DATA 0) itemoff 15005 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2314932224 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 12 key (384 INODE_ITEM 0) itemoff 14845 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 1572864 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 6 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 13 key (384 EXTENT_DATA 0) itemoff 14792 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2315194368 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 14 key (385 INODE_ITEM 0) itemoff 14632 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 15 key (385 EXTENT_DATA 0) itemoff 14579 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2315456512 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 16 key (386 INODE_ITEM 0) itemoff 14419 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 6553600 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 25 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 17 key (386 EXTENT_DATA 0) itemoff 14366 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2315988992 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 18 key (387 INODE_ITEM 0) itemoff 14206 itemsize 160 generation 802241 transid 802241 size 262144 nbytes 174587904 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 666 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1610592670.511339309 (2021-01-13 21:51:10) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 19 key (387 EXTENT_DATA 0) itemoff 14153 itemsize 53 generation 802241 type 1 (regular) extent data disk byte 11649187840 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 20 key (388 INODE_ITEM 0) itemoff 13993 itemsize 160 generation 765524 transid 765524 size 262144 nbytes 44040192 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 168 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609874507.916417349 (2021-01-05 14:21:47) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 21 key (388 EXTENT_DATA 0) itemoff 13940 itemsize 53 generation 765524 type 1 (regular) extent data disk byte 2318118912 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 22 key (389 INODE_ITEM 0) itemoff 13780 itemsize 160 generation 757140 transid 757140 size 262144 nbytes 9437184 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 36 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1609552308.382317581 (2021-01-01 20:51:48) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 23 key (389 EXTENT_DATA 0) itemoff 13727 itemsize 53 generation 757140 type 1 (regular) extent data disk byte 25752731648 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 24 key (390 INODE_ITEM 0) itemoff 13567 itemsize 160 generation 745597 transid 745597 size 262144 nbytes 5242880 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 20 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1608745812.850482769 (2020-12-23 12:50:12) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 25 key (390 EXTENT_DATA 0) itemoff 13514 itemsize 53 generation 745597 type 1 (regular) extent data disk byte 7834468352 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 26 key (391 INODE_ITEM 0) itemoff 13354 itemsize 160 generation 885295 transid 885295 size 262144 nbytes 477626368 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1822 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613083281.341313926 (2021-02-11 17:41:21) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 27 key (391 EXTENT_DATA 0) itemoff 13301 itemsize 53 generation 885295 type 1 (regular) extent data disk byte 56541450240 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 28 key (392 INODE_ITEM 0) itemoff 13141 itemsize 160 generation 885293 transid 885293 size 262144 nbytes 201064448 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 767 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613083215.522937734 (2021-02-11 17:40:15) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 29 key (392 EXTENT_DATA 0) itemoff 13088 itemsize 53 generation 885293 type 1 (regular) extent data disk byte 54390079488 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 30 key (394 INODE_ITEM 0) itemoff 12928 itemsize 160 generation 802243 transid 802243 size 262144 nbytes 67371008 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 257 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1610592741.153753253 (2021-01-13 21:52:21) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 31 key (394 EXTENT_DATA 0) itemoff 12875 itemsize 53 generation 802243 type 1 (regular) extent data disk byte 11432259584 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 32 key (395 INODE_ITEM 0) itemoff 12715 itemsize 160 generation 881621 transid 881621 size 262144 nbytes 272629760 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1040 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613048351.685578539 (2021-02-11 07:59:11) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 33 key (395 EXTENT_DATA 0) itemoff 12662 itemsize 53 generation 881621 type 1 (regular) extent data disk byte 12186574848 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 34 key (396 INODE_ITEM 0) itemoff 12502 itemsize 160 generation 888488 transid 888488 size 262144 nbytes 284688384 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1086 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613247217.331353729 (2021-02-13 15:13:37) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 35 key (396 EXTENT_DATA 0) itemoff 12449 itemsize 53 generation 888488 type 1 (regular) extent data disk byte 35193180160 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 36 key (397 INODE_ITEM 0) itemoff 12289 itemsize 160 generation 883273 transid 883273 size 262144 nbytes 269484032 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1028 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613076609.245318242 (2021-02-11 15:50:09) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 37 key (397 EXTENT_DATA 0) itemoff 12236 itemsize 53 generation 883273 type 1 (regular) extent data disk byte 29829939200 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 38 key (398 INODE_ITEM 0) itemoff 12076 itemsize 160 generation 883273 transid 883273 size 262144 nbytes 17039360 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 65 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 19:00:00) ctime 1613076609.247318252 (2021-02-11 15:50:09) mtime 0.0 (1969-12-31 19:00:00) otime 0.0 (1969-12-31 19:00:00) item 39 key (398 EXTENT_DATA 0) itemoff 12023 itemsize 53 generation 883273 type 1 (regular) extent data disk byte 29830201344 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 40 key (401 ROOT_ITEM 0) itemoff 11584 itemsize 439 generation 888895 root_dirid 256 bytenr 789610496 level 2 refs 1 lastsnap 0 byte_limit 0 bytes_used 521977856 flags 0x0(none) uuid c1b45a24-cf09-6e4c-9508-65a98db2aeaa ctransid 888895 otransid 757276 stransid 0 rtransid 0 ctime 1613320608.505109087 (2021-02-14 11:36:48) otime 1609553954.396338240 (2021-01-01 21:19:14) drop key (0 UNKNOWN.0 0) level 0 item 41 key (401 ROOT_BACKREF 5) itemoff 11560 itemsize 24 root backref key dirid 256 sequence 4 name root00 item 42 key (401 ROOT_REF 402) itemoff 11534 itemsize 26 root ref key dirid 178600 sequence 56 name machines item 43 key (402 ROOT_ITEM 0) itemoff 11095 itemsize 439 generation 888539 root_dirid 256 bytenr 134168576 level 0 refs 1 lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none) uuid 43fca145-ca33-6148-a049-50995f191549 ctransid 888539 otransid 758905 stransid 0 rtransid 0 ctime 1613273608.902470286 (2021-02-13 22:33:28) otime 1609556994.96448124 (2021-01-01 22:09:54) drop key (0 UNKNOWN.0 0) level 0 item 44 key (402 ROOT_BACKREF 401) itemoff 11069 itemsize 26 root backref key dirid 178600 sequence 56 name machines item 45 key (FREE_SPACE UNTYPED 30408704) itemoff 11028 itemsize 41 location key (257 INODE_ITEM 0) cache generation 888895 entries 492 bitmaps 8 item 46 key (FREE_SPACE UNTYPED 1104150528) itemoff 10987 itemsize 41 location key (259 INODE_ITEM 0) cache generation 888895 entries 1 bitmaps 0 item 47 key (FREE_SPACE UNTYPED 2177892352) itemoff 10946 itemsize 41 location key (260 INODE_ITEM 0) cache generation 888895 entries 8 bitmaps 0 item 48 key (FREE_SPACE UNTYPED 3251634176) itemoff 10905 itemsize 41 location key (261 INODE_ITEM 0) cache generation 888895 entries 6 bitmaps 0 item 49 key (FREE_SPACE UNTYPED 4325376000) itemoff 10864 itemsize 41 location key (262 INODE_ITEM 0) cache generation 888491 entries 0 bitmaps 0 item 50 key (FREE_SPACE UNTYPED 5399117824) itemoff 10823 itemsize 41 location key (263 INODE_ITEM 0) cache generation 888191 entries 0 bitmaps 0 item 51 key (FREE_SPACE UNTYPED 6472859648) itemoff 10782 itemsize 41 location key (264 INODE_ITEM 0) cache generation 888775 entries 0 bitmaps 0 item 52 key (FREE_SPACE UNTYPED 7546601472) itemoff 10741 itemsize 41 location key (265 INODE_ITEM 0) cache generation 888570 entries 1 bitmaps 0 item 53 key (FREE_SPACE UNTYPED 8620343296) itemoff 10700 itemsize 41 location key (267 INODE_ITEM 0) cache generation 888895 entries 28 bitmaps 0 item 54 key (FREE_SPACE UNTYPED 9694085120) itemoff 10659 itemsize 41 location key (268 INODE_ITEM 0) cache generation 888895 entries 5 bitmaps 0 item 55 key (FREE_SPACE UNTYPED 10767826944) itemoff 10618 itemsize 41 location key (269 INODE_ITEM 0) cache generation 888895 entries 3 bitmaps 0 item 56 key (FREE_SPACE UNTYPED 11841568768) itemoff 10577 itemsize 41 location key (270 INODE_ITEM 0) cache generation 888775 entries 1 bitmaps 0 item 57 key (FREE_SPACE UNTYPED 12915310592) itemoff 10536 itemsize 41 location key (271 INODE_ITEM 0) cache generation 888895 entries 33 bitmaps 0 item 58 key (FREE_SPACE UNTYPED 13989052416) itemoff 10495 itemsize 41 location key (272 INODE_ITEM 0) cache generation 888895 entries 34 bitmaps 0 item 59 key (FREE_SPACE UNTYPED 15062794240) itemoff 10454 itemsize 41 location key (273 INODE_ITEM 0) cache generation 888895 entries 27 bitmaps 0 item 60 key (FREE_SPACE UNTYPED 16136536064) itemoff 10413 itemsize 41 location key (274 INODE_ITEM 0) cache generation 888895 entries 7 bitmaps 0 item 61 key (FREE_SPACE UNTYPED 17210277888) itemoff 10372 itemsize 41 location key (275 INODE_ITEM 0) cache generation 888497 entries 0 bitmaps 0 item 62 key (FREE_SPACE UNTYPED 18284019712) itemoff 10331 itemsize 41 location key (276 INODE_ITEM 0) cache generation 888609 entries 3 bitmaps 0 item 63 key (FREE_SPACE UNTYPED 19357761536) itemoff 10290 itemsize 41 location key (277 INODE_ITEM 0) cache generation 888745 entries 44 bitmaps 0 item 64 key (FREE_SPACE UNTYPED 20431503360) itemoff 10249 itemsize 41 location key (278 INODE_ITEM 0) cache generation 888895 entries 15 bitmaps 0 item 65 key (FREE_SPACE UNTYPED 21505245184) itemoff 10208 itemsize 41 location key (280 INODE_ITEM 0) cache generation 888895 entries 21 bitmaps 0 item 66 key (FREE_SPACE UNTYPED 22578987008) itemoff 10167 itemsize 41 location key (281 INODE_ITEM 0) cache generation 888609 entries 9 bitmaps 0 item 67 key (FREE_SPACE UNTYPED 23652728832) itemoff 10126 itemsize 41 location key (282 INODE_ITEM 0) cache generation 888895 entries 9 bitmaps 0 item 68 key (FREE_SPACE UNTYPED 24726470656) itemoff 10085 itemsize 41 location key (283 INODE_ITEM 0) cache generation 888609 entries 1 bitmaps 0 item 69 key (FREE_SPACE UNTYPED 25800212480) itemoff 10044 itemsize 41 location key (284 INODE_ITEM 0) cache generation 888609 entries 4 bitmaps 0 item 70 key (FREE_SPACE UNTYPED 26873954304) itemoff 10003 itemsize 41 location key (285 INODE_ITEM 0) cache generation 888785 entries 2 bitmaps 0 item 71 key (FREE_SPACE UNTYPED 27947696128) itemoff 9962 itemsize 41 location key (286 INODE_ITEM 0) cache generation 888497 entries 0 bitmaps 0 item 72 key (FREE_SPACE UNTYPED 29021437952) itemoff 9921 itemsize 41 location key (287 INODE_ITEM 0) cache generation 888401 entries 0 bitmaps 0 item 73 key (FREE_SPACE UNTYPED 30095179776) itemoff 9880 itemsize 41 location key (288 INODE_ITEM 0) cache generation 888570 entries 2 bitmaps 0 item 74 key (FREE_SPACE UNTYPED 31168921600) itemoff 9839 itemsize 41 location key (289 INODE_ITEM 0) cache generation 888411 entries 0 bitmaps 0 item 75 key (FREE_SPACE UNTYPED 32242663424) itemoff 9798 itemsize 41 location key (290 INODE_ITEM 0) cache generation 888667 entries 2 bitmaps 0 item 76 key (FREE_SPACE UNTYPED 33316405248) itemoff 9757 itemsize 41 location key (291 INODE_ITEM 0) cache generation 888894 entries 6 bitmaps 0 item 77 key (FREE_SPACE UNTYPED 34390147072) itemoff 9716 itemsize 41 location key (292 INODE_ITEM 0) cache generation 888894 entries 51 bitmaps 0 item 78 key (FREE_SPACE UNTYPED 35463888896) itemoff 9675 itemsize 41 location key (293 INODE_ITEM 0) cache generation 888785 entries 7 bitmaps 0 item 79 key (FREE_SPACE UNTYPED 36537630720) itemoff 9634 itemsize 41 location key (294 INODE_ITEM 0) cache generation 888785 entries 64 bitmaps 0 item 80 key (FREE_SPACE UNTYPED 37611372544) itemoff 9593 itemsize 41 location key (295 INODE_ITEM 0) cache generation 888894 entries 411 bitmaps 8 item 81 key (FREE_SPACE UNTYPED 38685114368) itemoff 9552 itemsize 41 location key (297 INODE_ITEM 0) cache generation 873604 entries 0 bitmaps 0 item 82 key (FREE_SPACE UNTYPED 39758856192) itemoff 9511 itemsize 41 location key (298 INODE_ITEM 0) cache generation 888023 entries 0 bitmaps 0 item 83 key (FREE_SPACE UNTYPED 40832598016) itemoff 9470 itemsize 41 location key (299 INODE_ITEM 0) cache generation 886667 entries 0 bitmaps 0 item 84 key (FREE_SPACE UNTYPED 41906339840) itemoff 9429 itemsize 41 location key (300 INODE_ITEM 0) cache generation 888800 entries 66 bitmaps 0 item 85 key (FREE_SPACE UNTYPED 42980081664) itemoff 9388 itemsize 41 location key (301 INODE_ITEM 0) cache generation 888872 entries 21 bitmaps 0 item 86 key (FREE_SPACE UNTYPED 44053823488) itemoff 9347 itemsize 41 location key (302 INODE_ITEM 0) cache generation 888872 entries 12 bitmaps 0 item 87 key (FREE_SPACE UNTYPED 45127565312) itemoff 9306 itemsize 41 location key (303 INODE_ITEM 0) cache generation 888872 entries 165 bitmaps 7 item 88 key (FREE_SPACE UNTYPED 46201307136) itemoff 9265 itemsize 41 location key (306 INODE_ITEM 0) cache generation 888872 entries 71 bitmaps 0 item 89 key (FREE_SPACE UNTYPED 47275048960) itemoff 9224 itemsize 41 location key (307 INODE_ITEM 0) cache generation 885295 entries 1 bitmaps 0 item 90 key (FREE_SPACE UNTYPED 48348790784) itemoff 9183 itemsize 41 location key (308 INODE_ITEM 0) cache generation 888667 entries 198 bitmaps 1 item 91 key (FREE_SPACE UNTYPED 49422532608) itemoff 9142 itemsize 41 location key (309 INODE_ITEM 0) cache generation 888872 entries 139 bitmaps 1 item 92 key (FREE_SPACE UNTYPED 50496274432) itemoff 9101 itemsize 41 location key (310 INODE_ITEM 0) cache generation 888791 entries 136 bitmaps 0 item 93 key (FREE_SPACE UNTYPED 51570016256) itemoff 9060 itemsize 41 location key (311 INODE_ITEM 0) cache generation 888791 entries 108 bitmaps 1 item 94 key (FREE_SPACE UNTYPED 52643758080) itemoff 9019 itemsize 41 location key (312 INODE_ITEM 0) cache generation 885296 entries 129 bitmaps 1 item 95 key (FREE_SPACE UNTYPED 53717499904) itemoff 8978 itemsize 41 location key (313 INODE_ITEM 0) cache generation 888791 entries 296 bitmaps 8 item 96 key (FREE_SPACE UNTYPED 54791241728) itemoff 8937 itemsize 41 location key (314 INODE_ITEM 0) cache generation 888572 entries 84 bitmaps 0 item 97 key (FREE_SPACE UNTYPED 55864983552) itemoff 8896 itemsize 41 location key (315 INODE_ITEM 0) cache generation 888662 entries 246 bitmaps 6 item 98 key (FREE_SPACE UNTYPED 56938725376) itemoff 8855 itemsize 41 location key (316 INODE_ITEM 0) cache generation 885555 entries 147 bitmaps 2 item 99 key (FREE_SPACE UNTYPED 58012467200) itemoff 8814 itemsize 41 location key (317 INODE_ITEM 0) cache generation 888662 entries 110 bitmaps 1 item 100 key (FREE_SPACE UNTYPED 59086209024) itemoff 8773 itemsize 41 location key (318 INODE_ITEM 0) cache generation 887740 entries 178 bitmaps 6 item 101 key (FREE_SPACE UNTYPED 60159950848) itemoff 8732 itemsize 41 location key (319 INODE_ITEM 0) cache generation 887741 entries 18 bitmaps 0 item 102 key (FREE_SPACE UNTYPED 61233692672) itemoff 8691 itemsize 41 location key (320 INODE_ITEM 0) cache generation 888662 entries 136 bitmaps 1 item 103 key (FREE_SPACE UNTYPED 62307434496) itemoff 8650 itemsize 41 location key (321 INODE_ITEM 0) cache generation 886055 entries 133 bitmaps 1 item 104 key (FREE_SPACE UNTYPED 65528659968) itemoff 8609 itemsize 41 location key (324 INODE_ITEM 0) cache generation 885295 entries 41 bitmaps 0 item 105 key (FREE_SPACE UNTYPED 66602401792) itemoff 8568 itemsize 41 location key (325 INODE_ITEM 0) cache generation 888037 entries 37 bitmaps 1 item 106 key (FREE_SPACE UNTYPED 68749885440) itemoff 8527 itemsize 41 location key (327 INODE_ITEM 0) cache generation 886682 entries 31 bitmaps 0 item 107 key (FREE_SPACE UNTYPED 69823627264) itemoff 8486 itemsize 41 location key (328 INODE_ITEM 0) cache generation 765522 entries 6 bitmaps 0 item 108 key (FREE_SPACE UNTYPED 70897369088) itemoff 8445 itemsize 41 location key (330 INODE_ITEM 0) cache generation 888895 entries 416 bitmaps 8 item 109 key (FREE_SPACE UNTYPED 71971110912) itemoff 8404 itemsize 41 location key (331 INODE_ITEM 0) cache generation 888895 entries 425 bitmaps 8 item 110 key (FREE_SPACE UNTYPED 73044852736) itemoff 8363 itemsize 41 location key (332 INODE_ITEM 0) cache generation 888037 entries 40 bitmaps 0 item 111 key (FREE_SPACE UNTYPED 75192336384) itemoff 8322 itemsize 41 location key (334 INODE_ITEM 0) cache generation 885295 entries 102 bitmaps 1 item 112 key (FREE_SPACE UNTYPED 76266078208) itemoff 8281 itemsize 41 location key (335 INODE_ITEM 0) cache generation 888662 entries 121 bitmaps 0 item 113 key (FREE_SPACE UNTYPED 77339820032) itemoff 8240 itemsize 41 location key (336 INODE_ITEM 0) cache generation 887742 entries 131 bitmaps 1 item 114 key (FREE_SPACE UNTYPED 79487303680) itemoff 8199 itemsize 41 location key (338 INODE_ITEM 0) cache generation 888576 entries 105 bitmaps 0 item 115 key (FREE_SPACE UNTYPED 80561045504) itemoff 8158 itemsize 41 location key (339 INODE_ITEM 0) cache generation 885295 entries 61 bitmaps 0 item 116 key (FREE_SPACE UNTYPED 81634787328) itemoff 8117 itemsize 41 location key (340 INODE_ITEM 0) cache generation 885295 entries 56 bitmaps 0 item 117 key (FREE_SPACE UNTYPED 82708529152) itemoff 8076 itemsize 41 location key (341 INODE_ITEM 0) cache generation 743566 entries 1 bitmaps 0 item 118 key (FREE_SPACE UNTYPED 83782270976) itemoff 8035 itemsize 41 location key (342 INODE_ITEM 0) cache generation 885295 entries 231 bitmaps 5 item 119 key (FREE_SPACE UNTYPED 84856012800) itemoff 7994 itemsize 41 location key (343 INODE_ITEM 0) cache generation 888037 entries 125 bitmaps 1 item 120 key (FREE_SPACE UNTYPED 85929754624) itemoff 7953 itemsize 41 location key (344 INODE_ITEM 0) cache generation 888488 entries 218 bitmaps 5 item 121 key (FREE_SPACE UNTYPED 87003496448) itemoff 7912 itemsize 41 location key (346 INODE_ITEM 0) cache generation 888573 entries 28 bitmaps 0 item 122 key (FREE_SPACE UNTYPED 88077238272) itemoff 7871 itemsize 41 location key (349 INODE_ITEM 0) cache generation 885295 entries 21 bitmaps 0 item 123 key (FREE_SPACE UNTYPED 89150980096) itemoff 7830 itemsize 41 location key (350 INODE_ITEM 0) cache generation 885295 entries 20 bitmaps 0 item 124 key (FREE_SPACE UNTYPED 94519689216) itemoff 7789 itemsize 41 location key (355 INODE_ITEM 0) cache generation 765524 entries 379 bitmaps 0 item 125 key (FREE_SPACE UNTYPED 95593431040) itemoff 7748 itemsize 41 location key (356 INODE_ITEM 0) cache generation 885295 entries 1103 bitmaps 5 item 126 key (FREE_SPACE UNTYPED 96667172864) itemoff 7707 itemsize 41 location key (358 INODE_ITEM 0) cache generation 885295 entries 17 bitmaps 0 item 127 key (FREE_SPACE UNTYPED 97740914688) itemoff 7666 itemsize 41 location key (361 INODE_ITEM 0) cache generation 881621 entries 3 bitmaps 0 item 128 key (FREE_SPACE UNTYPED 98814656512) itemoff 7625 itemsize 41 location key (363 INODE_ITEM 0) cache generation 765524 entries 1 bitmaps 0 item 129 key (FREE_SPACE UNTYPED 99888398336) itemoff 7584 itemsize 41 location key (364 INODE_ITEM 0) cache generation 765524 entries 1 bitmaps 0 item 130 key (FREE_SPACE UNTYPED 100962140160) itemoff 7543 itemsize 41 location key (365 INODE_ITEM 0) cache generation 765524 entries 1 bitmaps 0 item 131 key (FREE_SPACE UNTYPED 102035881984) itemoff 7502 itemsize 41 location key (366 INODE_ITEM 0) cache generation 693323 entries 0 bitmaps 0 item 132 key (FREE_SPACE UNTYPED 103109623808) itemoff 7461 itemsize 41 location key (367 INODE_ITEM 0) cache generation 765524 entries 1 bitmaps 0 item 133 key (FREE_SPACE UNTYPED 104183365632) itemoff 7420 itemsize 41 location key (368 INODE_ITEM 0) cache generation 765524 entries 2 bitmaps 0 item 134 key (FREE_SPACE UNTYPED 105257107456) itemoff 7379 itemsize 41 location key (369 INODE_ITEM 0) cache generation 693327 entries 0 bitmaps 0 item 135 key (FREE_SPACE UNTYPED 106330849280) itemoff 7338 itemsize 41 location key (370 INODE_ITEM 0) cache generation 765524 entries 3 bitmaps 0 item 136 key (FREE_SPACE UNTYPED 107404591104) itemoff 7297 itemsize 41 location key (371 INODE_ITEM 0) cache generation 765524 entries 2 bitmaps 0 item 137 key (FREE_SPACE UNTYPED 108478332928) itemoff 7256 itemsize 41 location key (372 INODE_ITEM 0) cache generation 765524 entries 1 bitmaps 0 item 138 key (FREE_SPACE UNTYPED 109552074752) itemoff 7215 itemsize 41 location key (373 INODE_ITEM 0) cache generation 765524 entries 2 bitmaps 0 item 139 key (FREE_SPACE UNTYPED 110625816576) itemoff 7174 itemsize 41 location key (374 INODE_ITEM 0) cache generation 802241 entries 1 bitmaps 0 item 140 key (FREE_SPACE UNTYPED 111699558400) itemoff 7133 itemsize 41 location key (375 INODE_ITEM 0) cache generation 765524 entries 23 bitmaps 0 item 141 key (FREE_SPACE UNTYPED 112773300224) itemoff 7092 itemsize 41 location key (376 INODE_ITEM 0) cache generation 765524 entries 22 bitmaps 0 item 142 key (FREE_SPACE UNTYPED 113847042048) itemoff 7051 itemsize 41 location key (377 INODE_ITEM 0) cache generation 765524 entries 1 bitmaps 0 item 143 key (FREE_SPACE UNTYPED 114920783872) itemoff 7010 itemsize 41 location key (378 INODE_ITEM 0) cache generation 765524 entries 1 bitmaps 0 item 144 key (FREE_SPACE UNTYPED 115994525696) itemoff 6969 itemsize 41 location key (379 INODE_ITEM 0) cache generation 765524 entries 1 bitmaps 0 item 145 key (FREE_SPACE UNTYPED 117068267520) itemoff 6928 itemsize 41 location key (380 INODE_ITEM 0) cache generation 765524 entries 1 bitmaps 0 item 146 key (FREE_SPACE UNTYPED 118142009344) itemoff 6887 itemsize 41 location key (381 INODE_ITEM 0) cache generation 765524 entries 1 bitmaps 0 item 147 key (FREE_SPACE UNTYPED 119215751168) itemoff 6846 itemsize 41 location key (382 INODE_ITEM 0) cache generation 765524 entries 31 bitmaps 0 item 148 key (FREE_SPACE UNTYPED 120289492992) itemoff 6805 itemsize 41 location key (383 INODE_ITEM 0) cache generation 765524 entries 1 bitmaps 0 item 149 key (FREE_SPACE UNTYPED 121363234816) itemoff 6764 itemsize 41 location key (384 INODE_ITEM 0) cache generation 765524 entries 1 bitmaps 0 item 150 key (FREE_SPACE UNTYPED 122436976640) itemoff 6723 itemsize 41 location key (385 INODE_ITEM 0) cache generation 765524 entries 1 bitmaps 0 item 151 key (FREE_SPACE UNTYPED 123510718464) itemoff 6682 itemsize 41 location key (386 INODE_ITEM 0) cache generation 765524 entries 2 bitmaps 0 item 152 key (FREE_SPACE UNTYPED 124584460288) itemoff 6641 itemsize 41 location key (387 INODE_ITEM 0) cache generation 802241 entries 1 bitmaps 0 item 153 key (FREE_SPACE UNTYPED 125658202112) itemoff 6600 itemsize 41 location key (388 INODE_ITEM 0) cache generation 765524 entries 5 bitmaps 0 item 154 key (FREE_SPACE UNTYPED 126731943936) itemoff 6559 itemsize 41 location key (389 INODE_ITEM 0) cache generation 757140 entries 5 bitmaps 0 item 155 key (FREE_SPACE UNTYPED 127805685760) itemoff 6518 itemsize 41 location key (390 INODE_ITEM 0) cache generation 745597 entries 1 bitmaps 0 item 156 key (FREE_SPACE UNTYPED 128879427584) itemoff 6477 itemsize 41 location key (391 INODE_ITEM 0) cache generation 885295 entries 31 bitmaps 2 item 157 key (FREE_SPACE UNTYPED 129953169408) itemoff 6436 itemsize 41 location key (392 INODE_ITEM 0) cache generation 885293 entries 160 bitmaps 2 item 158 key (FREE_SPACE UNTYPED 132100653056) itemoff 6395 itemsize 41 location key (394 INODE_ITEM 0) cache generation 802243 entries 4 bitmaps 0 item 159 key (FREE_SPACE UNTYPED 133174394880) itemoff 6354 itemsize 41 location key (395 INODE_ITEM 0) cache generation 881621 entries 2 bitmaps 0 item 160 key (FREE_SPACE UNTYPED 134248136704) itemoff 6313 itemsize 41 location key (396 INODE_ITEM 0) cache generation 888488 entries 3 bitmaps 0 item 161 key (FREE_SPACE UNTYPED 135321878528) itemoff 6272 itemsize 41 location key (397 INODE_ITEM 0) cache generation 883273 entries 318 bitmaps 4 item 162 key (FREE_SPACE UNTYPED 136395620352) itemoff 6231 itemsize 41 location key (398 INODE_ITEM 0) cache generation 883273 entries 4 bitmaps 0 item 163 key (DATA_RELOC_TREE ROOT_ITEM 0) itemoff 5792 itemsize 439 generation 4 root_dirid 256 bytenr 30490624 level 0 refs 1 lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none) uuid 00000000-0000-0000-0000-000000000000 drop key (0 UNKNOWN.0 0) level 0 ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-03-09 21:06 ` Neal Gompa @ 2021-03-09 21:56 ` Josef Bacik 2021-03-09 23:31 ` Neal Gompa 0 siblings, 1 reply; 41+ messages in thread From: Josef Bacik @ 2021-03-09 21:56 UTC (permalink / raw) To: Neal Gompa; +Cc: Btrfs BTRFS On 3/9/21 4:06 PM, Neal Gompa wrote: > On Tue, Mar 9, 2021 at 2:04 PM Josef Bacik <josef@toxicpanda.com> wrote: >> >> On 3/8/21 8:12 PM, Neal Gompa wrote: >>> On Mon, Mar 8, 2021 at 5:04 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>> >>>> On 3/8/21 3:01 PM, Neal Gompa wrote: >>>>> On Mon, Mar 8, 2021 at 1:38 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>> >>>>>> On 3/5/21 8:03 PM, Neal Gompa wrote: >>>>>>> On Fri, Mar 5, 2021 at 5:01 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>> >>>>>>>> On 3/5/21 9:41 AM, Neal Gompa wrote: >>>>>>>>> On Fri, Mar 5, 2021 at 9:12 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>> >>>>>>>>>> On 3/4/21 6:54 PM, Neal Gompa wrote: >>>>>>>>>>> On Thu, Mar 4, 2021 at 3:25 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 3/3/21 2:38 PM, Neal Gompa wrote: >>>>>>>>>>>>> On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2/24/21 10:47 PM, Neal Gompa wrote: >>>>>>>>>>>>>>> On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2/24/21 9:23 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 2/22/21 11:03 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> journal: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can anyone help? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you try >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other magic. Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Josef >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> No dice... >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> build, and then run >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr >>>>>>>>>>>>>>>>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind >>>>>>>>>>>>>>>>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 >>>>>>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>>>>>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 >>>>>>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> It's better, but still no dice... :( >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. >>>>>>>>>>>>>>>>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Nope, I see this in the journal: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Ok git pull for-neal, rebuild, then run >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances >>>>>>>>>>>>>>>>>>>>>>>> that the level doesn't match??". Thanks, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Tried rescue mount again after running that and got a stack trace in >>>>>>>>>>>>>>>>>>>>>>> the kernel, detailed in the following attached log. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with >>>>>>>>>>>>>>>>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a >>>>>>>>>>>>>>>>>>>>>> kernel with a fix for this >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> https://paste.centos.org/view/7b48aaea >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> and see if that gets you further. Thanks, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>>>>>>>>>>>>>>>>>>>> Killed >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> The log from the journal is attached. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Ahh crud my bad, this should do it >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> https://paste.centos.org/view/ac2e61ef >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Patch doesn't apply (note it is patch 667 below): >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Ah sorry, should have just sent you an iterative patch. You can take the above >>>>>>>>>>>>>>>>>> patch and just delete the hunk from volumes.c as you already have that applied >>>>>>>>>>>>>>>>>> and then it'll work. Thanks, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Failed with a weird error...? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt >>>>>>>>>>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Journal log with traceback attached. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Last one maybe? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://paste.centos.org/view/80edd6fd >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Similar weird failure: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt >>>>>>>>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> No crash in the journal this time, though: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sorry Neal, you replied when I was in the middle of something and promptly >>>>>>>>>>>>>> forgot about it. I figured the fs root was fine, can you do the following so I >>>>>>>>>>>>>> can figure out from the error messages what might be wrong >>>>>>>>>>>>>> >>>>>>>>>>>>>> btrfs check --readonly >>>>>>>>>>>>>> btrfs restore -D >>>>>>>>>>>>>> btrfs restore -l >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> It didn't work.. Here's the output: >>>>>>>>>>>>> >>>>>>>>>>>>> [root@fedora ~]# btrfs check --readonly /dev/sdb3 >>>>>>>>>>>>> Opening filesystem to check... >>>>>>>>>>>>> ERROR: could not setup extent tree >>>>>>>>>>>>> ERROR: cannot open file system >>>>>>>>>>>>> [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt >>>>>>>>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>>>>>>>> Couldn't setup device tree >>>>>>>>>>>>> Could not open root, trying backup super >>>>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>>>>>> Ignoring transid failure >>>>>>>>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>>>>>>>> Couldn't setup device tree >>>>>>>>>>>>> Could not open root, trying backup super >>>>>>>>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 >>>>>>>>>>>>> Could not open root, trying backup super >>>>>>>>>>>>> [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt >>>>>>>>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>>>>>>>> Couldn't setup device tree >>>>>>>>>>>>> Could not open root, trying backup super >>>>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 >>>>>>>>>>>>> Ignoring transid failure >>>>>>>>>>>>> WARNING: could not setup extent tree, skipping it >>>>>>>>>>>>> Couldn't setup device tree >>>>>>>>>>>>> Could not open root, trying backup super >>>>>>>>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 >>>>>>>>>>>>> Could not open root, trying backup super >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hmm OK I think we want the neal magic for this one too, but before we go doing >>>>>>>>>>>> that can I get a >>>>>>>>>>>> >>>>>>>>>>>> btrfs inspect-internal -f /dev/whatever >>>>>>>>>>>> >>>>>>>>>>>> so I can make sure I'm not just blindly clobbering something. Thanks, >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Doesn't work, did you mean some other command? >>>>>>>>>>> >>>>>>>>>>> [root@fedora ~]# btrfs inspect-internal -f /dev/sdb3 >>>>>>>>>>> btrfs inspect-internal: unknown token '-f' >>>>>>>>>> >>>>>>>>>> Sigh, sorry, btrfs inspect-internal dump-super -f /dev/sdb3 >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> Ok I've pushed to the for-neal branch in my btrfs-progs, can you pull and make >>>>>>>> and then run >>>>>>>> >>>>>>>> ./btrfs-print-block /dev/sdb3 791281664 >>>>>>>> >>>>>>>> and capture everything it prints out? Thanks, >>>>>>>> >>>>>>> >>>>>>> Here's the output from the command. >>>>>>> >>>>>>> >>>>>> >>>>>> Hmm looks like the fs is offset a bit, can you do >>>>>> >>>>>> ./btrfs-print-block /dev/sdb3 799670272 >>>>>> >>>>> >>>>> This command caused my session to crash, but I do have a log of what >>>>> was captured before it crashed and attached it. >>>>> >>>>>> also while we're here can I get >>>>>> >>>>>> btrfs-find-root /dev/sdb3 >>>>>> >>>>> >>>>> This ran successfully and I've attached the output. >>>>> >>>> >>>> Ok we're going to try this again, and if it doesn't work it looks like your >>>> chunk root is ok, so I'll rig something up to make the translation work right, >>>> but for now lets do >>>> >>>> ./btrfs-print-block /dev/sdb3 792395776 >>>> >>> >>> I've attached the output from that command, which did run successfully. >>> >> >> Definitely need the translation, I pushed a new patch to the btrfs-progs branch >> for-neal. Pull, rebuild, and then run the same command again, hopefully this >> gives me what I want. Thanks, >> > > Done and attached the output. > > Ok, lets go with ./btrfs-neal-magic /dev/sdb3 792395776 888895 1 and see if that fixes everything. Thanks, Josef ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Recovering Btrfs from a freak failure of the disk controller 2021-03-09 21:56 ` Josef Bacik @ 2021-03-09 23:31 ` Neal Gompa 0 siblings, 0 replies; 41+ messages in thread From: Neal Gompa @ 2021-03-09 23:31 UTC (permalink / raw) To: Josef Bacik; +Cc: Btrfs BTRFS On Tue, Mar 9, 2021 at 4:56 PM Josef Bacik <josef@toxicpanda.com> wrote: > > On 3/9/21 4:06 PM, Neal Gompa wrote: > > On Tue, Mar 9, 2021 at 2:04 PM Josef Bacik <josef@toxicpanda.com> wrote: > >> > >> On 3/8/21 8:12 PM, Neal Gompa wrote: > >>> On Mon, Mar 8, 2021 at 5:04 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>> > >>>> On 3/8/21 3:01 PM, Neal Gompa wrote: > >>>>> On Mon, Mar 8, 2021 at 1:38 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>> > >>>>>> On 3/5/21 8:03 PM, Neal Gompa wrote: > >>>>>>> On Fri, Mar 5, 2021 at 5:01 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>> > >>>>>>>> On 3/5/21 9:41 AM, Neal Gompa wrote: > >>>>>>>>> On Fri, Mar 5, 2021 at 9:12 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>> > >>>>>>>>>> On 3/4/21 6:54 PM, Neal Gompa wrote: > >>>>>>>>>>> On Thu, Mar 4, 2021 at 3:25 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 3/3/21 2:38 PM, Neal Gompa wrote: > >>>>>>>>>>>>> On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 2/24/21 10:47 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>> On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 2/24/21 9:23 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 2/22/21 11:03 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik <josef@toxicpanda.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk controller failure > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs refuses to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors show up in the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> journal: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical (device sda3): corrupt leaf: root=401 block=796082176 slot=15 ino=203657, invalid inode transid: has 888896 expect [0, 888895] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): block=796082176 read time tree block corruption detected > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same issue. I can't > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do recovery in > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data off. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version 5.9.16 and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version 5.10.14. I'm > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can anyone help? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you try > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, and then the tree > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not we can work out some > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other magic. Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Josef > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> No dice... > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 found 888895 > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, just pull > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> build, and then run > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) the right bytenr > >>>>>>>>>>>>>>>>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from there. This is kind > >>>>>>>>>>>>>>>>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>>>>>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> It's better, but still no dice... :( > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which is annoying. > >>>>>>>>>>>>>>>>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is normal? Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Nope, I see this in the journal: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): enabling all of the rescue options > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring data csums > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): ignoring bad roots > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disabling log replay at mount time > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): has skinny extents > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): tree level mismatch detected, bytenr=791281664 level expected=1 has=2 > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device sda3): open_ctree failed > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Ok git pull for-neal, rebuild, then run > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, whats the chances > >>>>>>>>>>>>>>>>>>>>>>>> that the level doesn't match??". Thanks, > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Tried rescue mount again after running that and got a stack trace in > >>>>>>>>>>>>>>>>>>>>>>> the kernel, detailed in the following attached log. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only tested with > >>>>>>>>>>>>>>>>>>>>>> zero'ing the extent root and the csum root. You're going to have to build a > >>>>>>>>>>>>>>>>>>>>>> kernel with a fix for this > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> https://paste.centos.org/view/7b48aaea > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> and see if that gets you further. Thanks, > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>>>>>>>>>>>>>>>>>> Killed > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> The log from the journal is attached. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Ahh crud my bad, this should do it > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> https://paste.centos.org/view/ac2e61ef > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Patch doesn't apply (note it is patch 667 below): > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Ah sorry, should have just sent you an iterative patch. You can take the above > >>>>>>>>>>>>>>>>>> patch and just delete the hunk from volumes.c as you already have that applied > >>>>>>>>>>>>>>>>>> and then it'll work. Thanks, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Failed with a weird error...? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt > >>>>>>>>>>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Journal log with traceback attached. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Last one maybe? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> https://paste.centos.org/view/80edd6fd > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Similar weird failure: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>>>>>>>>>>>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> No crash in the journal this time, though: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of the rescue options > >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data csums > >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots > >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log replay at mount time > >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space caching is enabled > >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents > >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read fs tree: -2 > >>>>>>>>>>>>>>>> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Sorry Neal, you replied when I was in the middle of something and promptly > >>>>>>>>>>>>>> forgot about it. I figured the fs root was fine, can you do the following so I > >>>>>>>>>>>>>> can figure out from the error messages what might be wrong > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> btrfs check --readonly > >>>>>>>>>>>>>> btrfs restore -D > >>>>>>>>>>>>>> btrfs restore -l > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> It didn't work.. Here's the output: > >>>>>>>>>>>>> > >>>>>>>>>>>>> [root@fedora ~]# btrfs check --readonly /dev/sdb3 > >>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>> [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt > >>>>>>>>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>>>>>>>> Couldn't setup device tree > >>>>>>>>>>>>> Could not open root, trying backup super > >>>>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>>>>>> Ignoring transid failure > >>>>>>>>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>>>>>>>> Couldn't setup device tree > >>>>>>>>>>>>> Could not open root, trying backup super > >>>>>>>>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > >>>>>>>>>>>>> Could not open root, trying backup super > >>>>>>>>>>>>> [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt > >>>>>>>>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>>>>>>>> Couldn't setup device tree > >>>>>>>>>>>>> Could not open root, trying backup super > >>>>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>>>>>> parent transid verify failed on 796082176 wanted 888894 found 888896 > >>>>>>>>>>>>> Ignoring transid failure > >>>>>>>>>>>>> WARNING: could not setup extent tree, skipping it > >>>>>>>>>>>>> Couldn't setup device tree > >>>>>>>>>>>>> Could not open root, trying backup super > >>>>>>>>>>>>> ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 > >>>>>>>>>>>>> Could not open root, trying backup super > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Hmm OK I think we want the neal magic for this one too, but before we go doing > >>>>>>>>>>>> that can I get a > >>>>>>>>>>>> > >>>>>>>>>>>> btrfs inspect-internal -f /dev/whatever > >>>>>>>>>>>> > >>>>>>>>>>>> so I can make sure I'm not just blindly clobbering something. Thanks, > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Doesn't work, did you mean some other command? > >>>>>>>>>>> > >>>>>>>>>>> [root@fedora ~]# btrfs inspect-internal -f /dev/sdb3 > >>>>>>>>>>> btrfs inspect-internal: unknown token '-f' > >>>>>>>>>> > >>>>>>>>>> Sigh, sorry, btrfs inspect-internal dump-super -f /dev/sdb3 > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> Ok I've pushed to the for-neal branch in my btrfs-progs, can you pull and make > >>>>>>>> and then run > >>>>>>>> > >>>>>>>> ./btrfs-print-block /dev/sdb3 791281664 > >>>>>>>> > >>>>>>>> and capture everything it prints out? Thanks, > >>>>>>>> > >>>>>>> > >>>>>>> Here's the output from the command. > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> Hmm looks like the fs is offset a bit, can you do > >>>>>> > >>>>>> ./btrfs-print-block /dev/sdb3 799670272 > >>>>>> > >>>>> > >>>>> This command caused my session to crash, but I do have a log of what > >>>>> was captured before it crashed and attached it. > >>>>> > >>>>>> also while we're here can I get > >>>>>> > >>>>>> btrfs-find-root /dev/sdb3 > >>>>>> > >>>>> > >>>>> This ran successfully and I've attached the output. > >>>>> > >>>> > >>>> Ok we're going to try this again, and if it doesn't work it looks like your > >>>> chunk root is ok, so I'll rig something up to make the translation work right, > >>>> but for now lets do > >>>> > >>>> ./btrfs-print-block /dev/sdb3 792395776 > >>>> > >>> > >>> I've attached the output from that command, which did run successfully. > >>> > >> > >> Definitely need the translation, I pushed a new patch to the btrfs-progs branch > >> for-neal. Pull, rebuild, and then run the same command again, hopefully this > >> gives me what I want. Thanks, > >> > > > > Done and attached the output. > > > > > > Ok, lets go with > > ./btrfs-neal-magic /dev/sdb3 792395776 888895 1 > > and see if that fixes everything. Thanks, > That worked! I was able to mount the disk and recover pretty much all of my user data to a backup drive. Now I can set up a new machine with all my data again! :) Thanks for all the assistance, even though it was pretty crazy. Perhaps something good can come of this for making recovery easier in the future? -- 真実はいつも一つ！/ Always, there's only one truth! ^ permalink raw reply [flat|nested] 41+ messages in thread
end of thread, other threads:[~2021-03-09 23:35 UTC | newest] Thread overview: 41+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-02-14 20:25 Recovering Btrfs from a freak failure of the disk controller Neal Gompa 2021-02-14 21:33 ` Chris Murphy 2021-02-14 22:04 ` Chris Murphy 2021-02-14 22:11 ` Chris Murphy 2021-02-14 23:23 ` Neal Gompa 2021-02-14 23:51 ` Chris Murphy 2021-02-14 23:56 ` Neal Gompa 2021-02-16 15:19 ` Josef Bacik 2021-02-16 16:27 ` Neal Gompa 2021-02-16 18:11 ` Josef Bacik 2021-02-16 20:29 ` Neal Gompa 2021-02-16 21:24 ` Josef Bacik 2021-02-17 2:05 ` Neal Gompa 2021-02-17 14:36 ` Josef Bacik 2021-02-17 14:50 ` Neal Gompa 2021-02-17 14:59 ` Josef Bacik 2021-02-17 16:29 ` Neal Gompa 2021-02-17 16:44 ` Josef Bacik 2021-02-21 18:27 ` Neal Gompa 2021-02-22 19:34 ` Josef Bacik 2021-02-23 4:03 ` Neal Gompa 2021-02-23 15:05 ` Josef Bacik 2021-02-24 14:23 ` Neal Gompa 2021-02-24 15:44 ` Josef Bacik 2021-02-25 3:47 ` Neal Gompa 2021-03-03 18:42 ` Josef Bacik 2021-03-03 19:38 ` Neal Gompa 2021-03-04 20:25 ` Josef Bacik 2021-03-04 23:54 ` Neal Gompa 2021-03-05 14:12 ` Josef Bacik 2021-03-05 14:41 ` Neal Gompa 2021-03-05 22:01 ` Josef Bacik 2021-03-06 1:03 ` Neal Gompa 2021-03-08 18:38 ` Josef Bacik 2021-03-08 20:01 ` Neal Gompa 2021-03-08 22:04 ` Josef Bacik 2021-03-09 1:12 ` Neal Gompa 2021-03-09 19:04 ` Josef Bacik 2021-03-09 21:06 ` Neal Gompa 2021-03-09 21:56 ` Josef Bacik 2021-03-09 23:31 ` Neal Gompa
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.