* BTRFS fails mount after power failure
@ 2021-08-23 20:52 weldon
2021-08-23 23:54 ` Qu Wenruo
0 siblings, 1 reply; 6+ messages in thread
From: weldon @ 2021-08-23 20:52 UTC (permalink / raw)
To: linux-btrfs
Good day folks,
I awoke this morning to find that my UPS had died overnight and my Ubuntu
server with a 14.5TB (Raid 5) BTRFS volume went down with it. The machine
rebooted fine and the hardware reports no errors, however the BTRFS volume
will no longer mount. The OS boots fine, the 14.5TB volume is for data
storage only. gparted shows the volume/partition, and correctly reports
space used as well as total size. I've never encountered this type of issue
over the past year while using btrfs and I'm not sure where to start. A
number of google search results express caution when attempting to
recover/repair, so I'm hoping for some expert advice.
My dmesg log exceeds the 100,000 bytes restriction, so I'm unable to attach
it, so please ask if there's anything specific I can include otherwise.
# uname -a
Linux onyx 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC 2021
x86_64 x86_64 x86_64 GNU/Linux
# btrfs --version
btrfs-progs v5.4.1
# btrfs fi show
Label: 'Data' uuid: 7f500ee1-32b7-45a3-b1e9-deb7e1f59632
Total devices 1 FS bytes used 7.17TiB
devid 1 size 14.50TiB used 7.40TiB path /dev/sdb1
# dmesg | grep sdb
[ 2.312875] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
CAPACITY(16).
[ 2.313010] sd 32:0:1:0: [sdb] 31138512896 512-byte logical blocks: (15.9
TB/14.5 TiB)
[ 2.313062] sd 32:0:1:0: [sdb] Write Protect is off
[ 2.313065] sd 32:0:1:0: [sdb] Mode Sense: 61 00 00 00
[ 2.313116] sd 32:0:1:0: [sdb] Cache data unavailable
[ 2.313119] sd 32:0:1:0: [sdb] Assuming drive cache: write through
[ 2.333321] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
CAPACITY(16).
[ 2.396761] sdb: sdb1
[ 2.397170] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
CAPACITY(16).
[ 2.397261] sd 32:0:1:0: [sdb] Attached SCSI disk
[ 4.709963] BTRFS: device label Data devid 1 transid 120260 /dev/sdb1
[ 21.849570] BTRFS info (device sdb1): disk space caching is enabled
[ 21.849573] BTRFS info (device sdb1): has skinny extents
[ 22.023224] BTRFS error (device sdb1): parent transid verify failed on
7939752886272 wanted 120260 found 120262
[ 22.047940] BTRFS error (device sdb1): parent transid verify failed on
7939752886272 wanted 120260 found 120265
[ 22.047949] BTRFS warning (device sdb1): failed to read tree root
[ 22.089003] BTRFS error (device sdb1): open_ctree failed
root@onyx:/home/weldon# btrfs-find-root /dev/sdb1
parent transid verify failed on 7939752886272 wanted 120260 found 120262
parent transid verify failed on 7939752886272 wanted 120260 found 120265
parent transid verify failed on 7939752886272 wanted 120260 found 120265
Ignoring transid failure
WARNING: could not setup extent tree, skipping it
Couldn't setup device tree
Superblock thinks the generation is 120260
Superblock thinks the level is 1
Well block 7939758882816(gen: 120264 level: 1) seems good, but
generation/level doesn't match, want gen: 120260 level: 1
Well block 7939747938304(gen: 120263 level: 1) seems good, but
generation/level doesn't match, want gen: 120260 level: 1
Well block 7939756146688(gen: 120262 level: 1) seems good, but
generation/level doesn't match, want gen: 120260 level: 1
Well block 7939751559168(gen: 120261 level: 0) seems good, but
generation/level doesn't match, want gen: 120260 level: 1
*** A large selection of block references was removed due to character
count... if needed, I can resend with the full output.
Well block 1316967743488(gen: 1293 level: 0) seems good, but
generation/level doesn't match, want gen: 120260 level: 1
Well block 1316909662208(gen: 1283 level: 0) seems good, but
generation/level doesn't match, want gen: 120260 level: 1
Well block 1316908711936(gen: 1283 level: 0) seems good, but
generation/level doesn't match, want gen: 120260 level: 1
root@onyx:/home#
Any help or assistance would be greatly appreciated. Important data has
been backed up, however if it's possible to recover without thrashing the
entire volume, that would be preferred.
Regards,
Weldon
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: BTRFS fails mount after power failure
2021-08-23 20:52 BTRFS fails mount after power failure weldon
@ 2021-08-23 23:54 ` Qu Wenruo
2021-08-24 0:10 ` weldon
0 siblings, 1 reply; 6+ messages in thread
From: Qu Wenruo @ 2021-08-23 23:54 UTC (permalink / raw)
To: weldon, linux-btrfs
On 2021/8/24 上午4:52, weldon@newfietech.com wrote:
> Good day folks,
>
> I awoke this morning to find that my UPS had died overnight and my Ubuntu
> server with a 14.5TB (Raid 5) BTRFS volume went down with it.
RAID5 has known write hole bug, and although that bug won't cause
immediate problems, it slowly degrades the whole array with each
corrupted sector or unexpected power loss.
This would eventually bring down the array with enough degradation.
> The machine
> rebooted fine and the hardware reports no errors, however the BTRFS volume
> will no longer mount. The OS boots fine, the 14.5TB volume is for data
> storage only. gparted shows the volume/partition, and correctly reports
> space used as well as total size. I've never encountered this type of issue
> over the past year while using btrfs and I'm not sure where to start. A
> number of google search results express caution when attempting to
> recover/repair, so I'm hoping for some expert advice.
>
> My dmesg log exceeds the 100,000 bytes restriction, so I'm unable to attach
> it, so please ask if there's anything specific I can include otherwise.
>
> # uname -a
> Linux onyx 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC 2021
> x86_64 x86_64 x86_64 GNU/Linux
>
> # btrfs --version
> btrfs-progs v5.4.1
>
> # btrfs fi show
> Label: 'Data' uuid: 7f500ee1-32b7-45a3-b1e9-deb7e1f59632
> Total devices 1 FS bytes used 7.17TiB
> devid 1 size 14.50TiB used 7.40TiB path /dev/sdb1
>
> # dmesg | grep sdb
> [ 2.312875] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
> CAPACITY(16).
> [ 2.313010] sd 32:0:1:0: [sdb] 31138512896 512-byte logical blocks: (15.9
> TB/14.5 TiB)
> [ 2.313062] sd 32:0:1:0: [sdb] Write Protect is off
> [ 2.313065] sd 32:0:1:0: [sdb] Mode Sense: 61 00 00 00
> [ 2.313116] sd 32:0:1:0: [sdb] Cache data unavailable
> [ 2.313119] sd 32:0:1:0: [sdb] Assuming drive cache: write through
> [ 2.333321] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
> CAPACITY(16).
> [ 2.396761] sdb: sdb1
> [ 2.397170] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
> CAPACITY(16).
> [ 2.397261] sd 32:0:1:0: [sdb] Attached SCSI disk
> [ 4.709963] BTRFS: device label Data devid 1 transid 120260 /dev/sdb1
> [ 21.849570] BTRFS info (device sdb1): disk space caching is enabled
> [ 21.849573] BTRFS info (device sdb1): has skinny extents
> [ 22.023224] BTRFS error (device sdb1): parent transid verify failed on
> 7939752886272 wanted 120260 found 120262
> [ 22.047940] BTRFS error (device sdb1): parent transid verify failed on
> 7939752886272 wanted 120260 found 120265
This already shows some mismatch in on-disk data and recovered data from
parity.
This shows the on-disk data and parity have drifted from each other,
exactly the write hole problem.
Furthermore, the disk has newer data than what we expect.
What's the device model? It looks like a misbehavior, not sure if it's
from the hardware, or the btrfs code.
As RAID56 is already marked as unsafe for a while, not that much love
nor code fix is directed to RAID56, thus both cases are possible.
> [ 22.047949] BTRFS warning (device sdb1): failed to read tree root
> [ 22.089003] BTRFS error (device sdb1): open_ctree failed
>
> root@onyx:/home/weldon# btrfs-find-root /dev/sdb1
> parent transid verify failed on 7939752886272 wanted 120260 found 120262
> parent transid verify failed on 7939752886272 wanted 120260 found 120265
> parent transid verify failed on 7939752886272 wanted 120260 found 120265
> Ignoring transid failure
> WARNING: could not setup extent tree, skipping it
> Couldn't setup device tree
> Superblock thinks the generation is 120260
> Superblock thinks the level is 1
> Well block 7939758882816(gen: 120264 level: 1) seems good, but
> generation/level doesn't match, want gen: 120260 level: 1
> Well block 7939747938304(gen: 120263 level: 1) seems good, but
> generation/level doesn't match, want gen: 120260 level: 1
> Well block 7939756146688(gen: 120262 level: 1) seems good, but
> generation/level doesn't match, want gen: 120260 level: 1
> Well block 7939751559168(gen: 120261 level: 0) seems good, but
> generation/level doesn't match, want gen: 120260 level: 1
>
> *** A large selection of block references was removed due to character
> count... if needed, I can resend with the full output.
>
> Well block 1316967743488(gen: 1293 level: 0) seems good, but
> generation/level doesn't match, want gen: 120260 level: 1
> Well block 1316909662208(gen: 1283 level: 0) seems good, but
> generation/level doesn't match, want gen: 120260 level: 1
> Well block 1316908711936(gen: 1283 level: 0) seems good, but
> generation/level doesn't match, want gen: 120260 level: 1
> root@onyx:/home#
>
> Any help or assistance would be greatly appreciated. Important data has
> been backed up, however if it's possible to recover without thrashing the
> entire volume, that would be preferred.
First thing first, don't expect too much about magically turning the fs
back to fully functional status.
Transid error is always tricky for btrfs.
But for your case, I'm guessing your sdb1 does not have the latest super
block.
We have newer tree roots on disk, but older super block.
Maybe you would like to try "btrfs ins dump-tree" on all the involved
disks, and find if there is newer super blocks.
Thanks,
Qu
>
> Regards,
> Weldon
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: BTRFS fails mount after power failure
2021-08-23 23:54 ` Qu Wenruo
@ 2021-08-24 0:10 ` weldon
2021-08-24 0:38 ` Qu Wenruo
0 siblings, 1 reply; 6+ messages in thread
From: weldon @ 2021-08-24 0:10 UTC (permalink / raw)
To: 'Qu Wenruo', linux-btrfs
Thank you for the reply Qu.
The hardware setup is a bit wonky in a home lab, but is as follows:
Dell PowerEdge R510 Chassis
Dell PERC H700
6 * 4TB SATA Disks in a RAID 5 configuration
ESXi 6.5 hypervisor sees storage as local DELL Disk, 18.19TB
17.66TB Provisioned as a Datastore on the hypervisor, VMFS5.
- 14.5TB provisioned as a vmdk and presented as local disk to Ubuntu virtual machine, mounted as /data (btrfs)
- 200GB provisioned as vmdk and presented as local disk to Ubuntu virtual machine, mounted as / (ext4)
Happy and willing to try any suggestions you may have.
root@onyx:/home# btrfs ins dump-tree /dev/sdb1
btrfs-progs v5.4.1
parent transid verify failed on 7939752886272 wanted 120260 found 120262
parent transid verify failed on 7939752886272 wanted 120260 found 120265
parent transid verify failed on 7939752886272 wanted 120260 found 120265
Ignoring transid failure
WARNING: could not setup extent tree, skipping it
Couldn't setup device tree
ERROR: unable to open /dev/sdb1
root@onyx:/home#
Thanks in advance,
Weldon
-----Original Message-----
From: Qu Wenruo <quwenruo.btrfs@gmx.com>
Sent: August 23, 2021 5:55 PM
To: weldon@newfietech.com; linux-btrfs@vger.kernel.org
Subject: Re: BTRFS fails mount after power failure
On 2021/8/24 上午4:52, weldon@newfietech.com wrote:
> Good day folks,
>
> I awoke this morning to find that my UPS had died overnight and my
> Ubuntu server with a 14.5TB (Raid 5) BTRFS volume went down with it.
RAID5 has known write hole bug, and although that bug won't cause immediate problems, it slowly degrades the whole array with each corrupted sector or unexpected power loss.
This would eventually bring down the array with enough degradation.
> The machine
> rebooted fine and the hardware reports no errors, however the BTRFS
>volume will no longer mount. The OS boots fine, the 14.5TB volume is
>for data storage only. gparted shows the volume/partition, and
>correctly reports space used as well as total size. I've never
>encountered this type of issue over the past year while using btrfs
>and I'm not sure where to start. A number of google search results
>express caution when attempting to recover/repair, so I'm hoping for some expert advice.
>
> My dmesg log exceeds the 100,000 bytes restriction, so I'm unable to
> attach it, so please ask if there's anything specific I can include otherwise.
>
> # uname -a
> Linux onyx 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC
> 2021
> x86_64 x86_64 x86_64 GNU/Linux
>
> # btrfs --version
> btrfs-progs v5.4.1
>
> # btrfs fi show
> Label: 'Data' uuid: 7f500ee1-32b7-45a3-b1e9-deb7e1f59632
> Total devices 1 FS bytes used 7.17TiB
> devid 1 size 14.50TiB used 7.40TiB path /dev/sdb1
>
> # dmesg | grep sdb
> [ 2.312875] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
> CAPACITY(16).
> [ 2.313010] sd 32:0:1:0: [sdb] 31138512896 512-byte logical blocks: (15.9
> TB/14.5 TiB)
> [ 2.313062] sd 32:0:1:0: [sdb] Write Protect is off
> [ 2.313065] sd 32:0:1:0: [sdb] Mode Sense: 61 00 00 00
> [ 2.313116] sd 32:0:1:0: [sdb] Cache data unavailable
> [ 2.313119] sd 32:0:1:0: [sdb] Assuming drive cache: write through
> [ 2.333321] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
> CAPACITY(16).
> [ 2.396761] sdb: sdb1
> [ 2.397170] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
> CAPACITY(16).
> [ 2.397261] sd 32:0:1:0: [sdb] Attached SCSI disk
> [ 4.709963] BTRFS: device label Data devid 1 transid 120260 /dev/sdb1
> [ 21.849570] BTRFS info (device sdb1): disk space caching is enabled
> [ 21.849573] BTRFS info (device sdb1): has skinny extents
> [ 22.023224] BTRFS error (device sdb1): parent transid verify failed on
> 7939752886272 wanted 120260 found 120262
> [ 22.047940] BTRFS error (device sdb1): parent transid verify failed on
> 7939752886272 wanted 120260 found 120265
This already shows some mismatch in on-disk data and recovered data from parity.
This shows the on-disk data and parity have drifted from each other, exactly the write hole problem.
Furthermore, the disk has newer data than what we expect.
What's the device model? It looks like a misbehavior, not sure if it's from the hardware, or the btrfs code.
As RAID56 is already marked as unsafe for a while, not that much love nor code fix is directed to RAID56, thus both cases are possible.
> [ 22.047949] BTRFS warning (device sdb1): failed to read tree root
> [ 22.089003] BTRFS error (device sdb1): open_ctree failed
>
> root@onyx:/home/weldon# btrfs-find-root /dev/sdb1 parent transid
> verify failed on 7939752886272 wanted 120260 found 120262 parent
> transid verify failed on 7939752886272 wanted 120260 found 120265
> parent transid verify failed on 7939752886272 wanted 120260 found
> 120265 Ignoring transid failure
> WARNING: could not setup extent tree, skipping it Couldn't setup
> device tree Superblock thinks the generation is 120260 Superblock
> thinks the level is 1 Well block 7939758882816(gen: 120264 level: 1)
> seems good, but generation/level doesn't match, want gen: 120260
> level: 1 Well block 7939747938304(gen: 120263 level: 1) seems good,
> but generation/level doesn't match, want gen: 120260 level: 1 Well
> block 7939756146688(gen: 120262 level: 1) seems good, but
> generation/level doesn't match, want gen: 120260 level: 1 Well block
> 7939751559168(gen: 120261 level: 0) seems good, but generation/level
> doesn't match, want gen: 120260 level: 1
>
> *** A large selection of block references was removed due to character
> count... if needed, I can resend with the full output.
>
> Well block 1316967743488(gen: 1293 level: 0) seems good, but
> generation/level doesn't match, want gen: 120260 level: 1 Well block
> 1316909662208(gen: 1283 level: 0) seems good, but generation/level
> doesn't match, want gen: 120260 level: 1 Well block 1316908711936(gen:
> 1283 level: 0) seems good, but generation/level doesn't match, want
> gen: 120260 level: 1 root@onyx:/home#
>
> Any help or assistance would be greatly appreciated. Important data
> has been backed up, however if it's possible to recover without
> thrashing the entire volume, that would be preferred.
First thing first, don't expect too much about magically turning the fs back to fully functional status.
Transid error is always tricky for btrfs.
But for your case, I'm guessing your sdb1 does not have the latest super block.
We have newer tree roots on disk, but older super block.
Maybe you would like to try "btrfs ins dump-tree" on all the involved disks, and find if there is newer super blocks.
Thanks,
Qu
>
> Regards,
> Weldon
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: BTRFS fails mount after power failure
2021-08-24 0:10 ` weldon
@ 2021-08-24 0:38 ` Qu Wenruo
2021-08-24 0:52 ` weldon
0 siblings, 1 reply; 6+ messages in thread
From: Qu Wenruo @ 2021-08-24 0:38 UTC (permalink / raw)
To: weldon, linux-btrfs
On 2021/8/24 上午8:10, weldon@newfietech.com wrote:
> Thank you for the reply Qu.
>
> The hardware setup is a bit wonky in a home lab, but is as follows:
>
> Dell PowerEdge R510 Chassis
> Dell PERC H700
> 6 * 4TB SATA Disks in a RAID 5 configuration
The RAID5 is not provided by btrfs, but some hardware RAID controller?
Then we don't need to bother the btrfs RAID5 bug.
But still, this means the RAID controller or the hdd is not doing proper
flush/fua.
This means, next time your UPS went down or a kernel crash happens, you
may still hit a similar problem.
And this time, we're pretty sure it's less possible to blame btrfs code.
> ESXi 6.5 hypervisor sees storage as local DELL Disk, 18.19TB
>
> 17.66TB Provisioned as a Datastore on the hypervisor, VMFS5.
> - 14.5TB provisioned as a vmdk and presented as local disk to Ubuntu virtual machine, mounted as /data (btrfs)
> - 200GB provisioned as vmdk and presented as local disk to Ubuntu virtual machine, mounted as / (ext4)
>
> Happy and willing to try any suggestions you may have.
>
> root@onyx:/home# btrfs ins dump-tree /dev/sdb1
My bad, I mean "btrfs ins dump-super -fFa", but that's for the case of
btrfs RAID5 setup.
Since you're using hardware RAID5 controller, we can go direct to the
recovery part.
Your previous find-root output would be pretty helpful.
You can try btrfs-check with -r option:
# btrfs check -r 7939758882816 /dev/sdb1
To see how many errors it throws. if it had almost no error, then it has
a pretty high chance to recover the data.
You can also try other bytenr from your find-root output, but I guess
you only need to try the first 4 bytenrs.
Thanks,
Qu
> btrfs-progs v5.4.1
> parent transid verify failed on 7939752886272 wanted 120260 found 120262
> parent transid verify failed on 7939752886272 wanted 120260 found 120265
> parent transid verify failed on 7939752886272 wanted 120260 found 120265
> Ignoring transid failure
> WARNING: could not setup extent tree, skipping it
> Couldn't setup device tree
> ERROR: unable to open /dev/sdb1
> root@onyx:/home#
>
>
> Thanks in advance,
> Weldon
>
>
> -----Original Message-----
> From: Qu Wenruo <quwenruo.btrfs@gmx.com>
> Sent: August 23, 2021 5:55 PM
> To: weldon@newfietech.com; linux-btrfs@vger.kernel.org
> Subject: Re: BTRFS fails mount after power failure
>
>
>
> On 2021/8/24 上午4:52, weldon@newfietech.com wrote:
>> Good day folks,
>>
>> I awoke this morning to find that my UPS had died overnight and my
>> Ubuntu server with a 14.5TB (Raid 5) BTRFS volume went down with it.
>
> RAID5 has known write hole bug, and although that bug won't cause immediate problems, it slowly degrades the whole array with each corrupted sector or unexpected power loss.
>
> This would eventually bring down the array with enough degradation.
>
>> The machine
>> rebooted fine and the hardware reports no errors, however the BTRFS
>> volume will no longer mount. The OS boots fine, the 14.5TB volume is
>> for data storage only. gparted shows the volume/partition, and
>> correctly reports space used as well as total size. I've never
>> encountered this type of issue over the past year while using btrfs
>> and I'm not sure where to start. A number of google search results
>> express caution when attempting to recover/repair, so I'm hoping for some expert advice.
>>
>> My dmesg log exceeds the 100,000 bytes restriction, so I'm unable to
>> attach it, so please ask if there's anything specific I can include otherwise.
>>
>> # uname -a
>> Linux onyx 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC
>> 2021
>> x86_64 x86_64 x86_64 GNU/Linux
>>
>> # btrfs --version
>> btrfs-progs v5.4.1
>>
>> # btrfs fi show
>> Label: 'Data' uuid: 7f500ee1-32b7-45a3-b1e9-deb7e1f59632
>> Total devices 1 FS bytes used 7.17TiB
>> devid 1 size 14.50TiB used 7.40TiB path /dev/sdb1
>>
>> # dmesg | grep sdb
>> [ 2.312875] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
>> CAPACITY(16).
>> [ 2.313010] sd 32:0:1:0: [sdb] 31138512896 512-byte logical blocks: (15.9
>> TB/14.5 TiB)
>> [ 2.313062] sd 32:0:1:0: [sdb] Write Protect is off
>> [ 2.313065] sd 32:0:1:0: [sdb] Mode Sense: 61 00 00 00
>> [ 2.313116] sd 32:0:1:0: [sdb] Cache data unavailable
>> [ 2.313119] sd 32:0:1:0: [sdb] Assuming drive cache: write through
>> [ 2.333321] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
>> CAPACITY(16).
>> [ 2.396761] sdb: sdb1
>> [ 2.397170] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
>> CAPACITY(16).
>> [ 2.397261] sd 32:0:1:0: [sdb] Attached SCSI disk
>> [ 4.709963] BTRFS: device label Data devid 1 transid 120260 /dev/sdb1
>> [ 21.849570] BTRFS info (device sdb1): disk space caching is enabled
>> [ 21.849573] BTRFS info (device sdb1): has skinny extents
>> [ 22.023224] BTRFS error (device sdb1): parent transid verify failed on
>> 7939752886272 wanted 120260 found 120262
>> [ 22.047940] BTRFS error (device sdb1): parent transid verify failed on
>> 7939752886272 wanted 120260 found 120265
>
> This already shows some mismatch in on-disk data and recovered data from parity.
>
> This shows the on-disk data and parity have drifted from each other, exactly the write hole problem.
>
> Furthermore, the disk has newer data than what we expect.
>
> What's the device model? It looks like a misbehavior, not sure if it's from the hardware, or the btrfs code.
> As RAID56 is already marked as unsafe for a while, not that much love nor code fix is directed to RAID56, thus both cases are possible.
>
>> [ 22.047949] BTRFS warning (device sdb1): failed to read tree root
>> [ 22.089003] BTRFS error (device sdb1): open_ctree failed
>>
>> root@onyx:/home/weldon# btrfs-find-root /dev/sdb1 parent transid
>> verify failed on 7939752886272 wanted 120260 found 120262 parent
>> transid verify failed on 7939752886272 wanted 120260 found 120265
>> parent transid verify failed on 7939752886272 wanted 120260 found
>> 120265 Ignoring transid failure
>> WARNING: could not setup extent tree, skipping it Couldn't setup
>> device tree Superblock thinks the generation is 120260 Superblock
>> thinks the level is 1 Well block 7939758882816(gen: 120264 level: 1)
>> seems good, but generation/level doesn't match, want gen: 120260
>> level: 1 Well block 7939747938304(gen: 120263 level: 1) seems good,
>> but generation/level doesn't match, want gen: 120260 level: 1 Well
>> block 7939756146688(gen: 120262 level: 1) seems good, but
>> generation/level doesn't match, want gen: 120260 level: 1 Well block
>> 7939751559168(gen: 120261 level: 0) seems good, but generation/level
>> doesn't match, want gen: 120260 level: 1
>>
>> *** A large selection of block references was removed due to character
>> count... if needed, I can resend with the full output.
>>
>> Well block 1316967743488(gen: 1293 level: 0) seems good, but
>> generation/level doesn't match, want gen: 120260 level: 1 Well block
>> 1316909662208(gen: 1283 level: 0) seems good, but generation/level
>> doesn't match, want gen: 120260 level: 1 Well block 1316908711936(gen:
>> 1283 level: 0) seems good, but generation/level doesn't match, want
>> gen: 120260 level: 1 root@onyx:/home#
>>
>> Any help or assistance would be greatly appreciated. Important data
>> has been backed up, however if it's possible to recover without
>> thrashing the entire volume, that would be preferred.
>
> First thing first, don't expect too much about magically turning the fs back to fully functional status.
> Transid error is always tricky for btrfs.
>
>
> But for your case, I'm guessing your sdb1 does not have the latest super block.
> We have newer tree roots on disk, but older super block.
>
> Maybe you would like to try "btrfs ins dump-tree" on all the involved disks, and find if there is newer super blocks.
>
> Thanks,
> Qu
>>
>> Regards,
>> Weldon
>>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: BTRFS fails mount after power failure
2021-08-24 0:38 ` Qu Wenruo
@ 2021-08-24 0:52 ` weldon
2021-08-24 1:54 ` Qu Wenruo
0 siblings, 1 reply; 6+ messages in thread
From: weldon @ 2021-08-24 0:52 UTC (permalink / raw)
To: 'Qu Wenruo', linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 10824 bytes --]
I've attached the complete results of the btrfs-find-root command.
If I understood your directions correctly, below is the result:
root@onyx:/home# btrfs check -r 7939758882816 /dev/sdb1
Opening filesystem to check...
parent transid verify failed on 7939758882816 wanted 120260 found 120264
parent transid verify failed on 7939758882816 wanted 120260 found 120264
parent transid verify failed on 7939758882816 wanted 120260 found 120264
Ignoring transid failure
parent transid verify failed on 7939751723008 wanted 120264 found 120262
parent transid verify failed on 7939751723008 wanted 120264 found 120266
parent transid verify failed on 7939751723008 wanted 120264 found 120266
Ignoring transid failure
parent transid verify failed on 7939735683072 wanted 120264 found 120263
parent transid verify failed on 7939735683072 wanted 120261 found 120264
Ignoring transid failure
parent transid verify failed on 7939734437888 wanted 120264 found 120253
Checking filesystem on /dev/sdb1
UUID: 7f500ee1-32b7-45a3-b1e9-deb7e1f59632
[1/7] checking root items
Error: could not find extent items for root 18446744073709551607
ERROR: failed to repair root items: No such file or directory
root@onyx:/home#
root@onyx:/home# btrfs check -r 7939747938304 /dev/sdb1
Opening filesystem to check...
parent transid verify failed on 7939747938304 wanted 120260 found 120263
parent transid verify failed on 7939747938304 wanted 120260 found 120265
parent transid verify failed on 7939747938304 wanted 120260 found 120265
Ignoring transid failure
ERROR: could not setup extent tree
ERROR: cannot open file system
root@onyx:/home#
root@onyx:/home# btrfs check -r 7939756146688 /dev/sdb1
Opening filesystem to check...
parent transid verify failed on 7939756146688 wanted 120260 found 120262
parent transid verify failed on 7939756146688 wanted 120260 found 120264
parent transid verify failed on 7939756146688 wanted 120260 found 120264
Ignoring transid failure
ERROR: could not setup extent tree
ERROR: cannot open file system
root@onyx:/home#
root@onyx:/home# btrfs check -r 7939751559168 /dev/sdb1
Opening filesystem to check...
parent transid verify failed on 7939751559168 wanted 120260 found 120261
parent transid verify failed on 7939751559168 wanted 120260 found 120261
parent transid verify failed on 7939751559168 wanted 120260 found 120261
Ignoring transid failure
ERROR: could not setup extent tree
ERROR: cannot open file system
root@onyx:/home#
Thanks again,
Weldon
-----Original Message-----
From: Qu Wenruo <quwenruo.btrfs@gmx.com>
Sent: August 23, 2021 6:39 PM
To: weldon@newfietech.com; linux-btrfs@vger.kernel.org
Subject: Re: BTRFS fails mount after power failure
On 2021/8/24 上午8:10, weldon@newfietech.com wrote:
> Thank you for the reply Qu.
>
> The hardware setup is a bit wonky in a home lab, but is as follows:
>
> Dell PowerEdge R510 Chassis
> Dell PERC H700
> 6 * 4TB SATA Disks in a RAID 5 configuration
The RAID5 is not provided by btrfs, but some hardware RAID controller?
Then we don't need to bother the btrfs RAID5 bug.
But still, this means the RAID controller or the hdd is not doing proper flush/fua.
This means, next time your UPS went down or a kernel crash happens, you may still hit a similar problem.
And this time, we're pretty sure it's less possible to blame btrfs code.
> ESXi 6.5 hypervisor sees storage as local DELL Disk, 18.19TB
>
> 17.66TB Provisioned as a Datastore on the hypervisor, VMFS5.
> - 14.5TB provisioned as a vmdk and presented as local disk to Ubuntu
> virtual machine, mounted as /data (btrfs)
> - 200GB provisioned as vmdk and presented as local disk to Ubuntu
> virtual machine, mounted as / (ext4)
>
> Happy and willing to try any suggestions you may have.
>
> root@onyx:/home# btrfs ins dump-tree /dev/sdb1
My bad, I mean "btrfs ins dump-super -fFa", but that's for the case of btrfs RAID5 setup.
Since you're using hardware RAID5 controller, we can go direct to the recovery part.
Your previous find-root output would be pretty helpful.
You can try btrfs-check with -r option:
# btrfs check -r 7939758882816 /dev/sdb1
To see how many errors it throws. if it had almost no error, then it has a pretty high chance to recover the data.
You can also try other bytenr from your find-root output, but I guess you only need to try the first 4 bytenrs.
Thanks,
Qu
> btrfs-progs v5.4.1
> parent transid verify failed on 7939752886272 wanted 120260 found
> 120262 parent transid verify failed on 7939752886272 wanted 120260
> found 120265 parent transid verify failed on 7939752886272 wanted
> 120260 found 120265 Ignoring transid failure
> WARNING: could not setup extent tree, skipping it Couldn't setup
> device tree
> ERROR: unable to open /dev/sdb1
> root@onyx:/home#
>
>
> Thanks in advance,
> Weldon
>
>
> -----Original Message-----
> From: Qu Wenruo <quwenruo.btrfs@gmx.com>
> Sent: August 23, 2021 5:55 PM
> To: weldon@newfietech.com; linux-btrfs@vger.kernel.org
> Subject: Re: BTRFS fails mount after power failure
>
>
>
> On 2021/8/24 上午4:52, weldon@newfietech.com wrote:
>> Good day folks,
>>
>> I awoke this morning to find that my UPS had died overnight and my
>> Ubuntu server with a 14.5TB (Raid 5) BTRFS volume went down with it.
>
> RAID5 has known write hole bug, and although that bug won't cause immediate problems, it slowly degrades the whole array with each corrupted sector or unexpected power loss.
>
> This would eventually bring down the array with enough degradation.
>
>> The machine
>> rebooted fine and the hardware reports no errors, however the BTRFS
>> volume will no longer mount. The OS boots fine, the 14.5TB volume
>> is for data storage only. gparted shows the volume/partition, and
>> correctly reports space used as well as total size. I've never
>> encountered this type of issue over the past year while using btrfs
>> and I'm not sure where to start. A number of google search results
>> express caution when attempting to recover/repair, so I'm hoping for some expert advice.
>>
>> My dmesg log exceeds the 100,000 bytes restriction, so I'm unable to
>> attach it, so please ask if there's anything specific I can include otherwise.
>>
>> # uname -a
>> Linux onyx 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC
>> 2021
>> x86_64 x86_64 x86_64 GNU/Linux
>>
>> # btrfs --version
>> btrfs-progs v5.4.1
>>
>> # btrfs fi show
>> Label: 'Data' uuid: 7f500ee1-32b7-45a3-b1e9-deb7e1f59632
>> Total devices 1 FS bytes used 7.17TiB
>> devid 1 size 14.50TiB used 7.40TiB path /dev/sdb1
>>
>> # dmesg | grep sdb
>> [ 2.312875] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
>> CAPACITY(16).
>> [ 2.313010] sd 32:0:1:0: [sdb] 31138512896 512-byte logical blocks: (15.9
>> TB/14.5 TiB)
>> [ 2.313062] sd 32:0:1:0: [sdb] Write Protect is off
>> [ 2.313065] sd 32:0:1:0: [sdb] Mode Sense: 61 00 00 00
>> [ 2.313116] sd 32:0:1:0: [sdb] Cache data unavailable
>> [ 2.313119] sd 32:0:1:0: [sdb] Assuming drive cache: write through
>> [ 2.333321] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
>> CAPACITY(16).
>> [ 2.396761] sdb: sdb1
>> [ 2.397170] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
>> CAPACITY(16).
>> [ 2.397261] sd 32:0:1:0: [sdb] Attached SCSI disk
>> [ 4.709963] BTRFS: device label Data devid 1 transid 120260 /dev/sdb1
>> [ 21.849570] BTRFS info (device sdb1): disk space caching is enabled
>> [ 21.849573] BTRFS info (device sdb1): has skinny extents
>> [ 22.023224] BTRFS error (device sdb1): parent transid verify failed on
>> 7939752886272 wanted 120260 found 120262
>> [ 22.047940] BTRFS error (device sdb1): parent transid verify failed on
>> 7939752886272 wanted 120260 found 120265
>
> This already shows some mismatch in on-disk data and recovered data from parity.
>
> This shows the on-disk data and parity have drifted from each other, exactly the write hole problem.
>
> Furthermore, the disk has newer data than what we expect.
>
> What's the device model? It looks like a misbehavior, not sure if it's from the hardware, or the btrfs code.
> As RAID56 is already marked as unsafe for a while, not that much love nor code fix is directed to RAID56, thus both cases are possible.
>
>> [ 22.047949] BTRFS warning (device sdb1): failed to read tree root
>> [ 22.089003] BTRFS error (device sdb1): open_ctree failed
>>
>> root@onyx:/home/weldon# btrfs-find-root /dev/sdb1 parent transid
>> verify failed on 7939752886272 wanted 120260 found 120262 parent
>> transid verify failed on 7939752886272 wanted 120260 found 120265
>> parent transid verify failed on 7939752886272 wanted 120260 found
>> 120265 Ignoring transid failure
>> WARNING: could not setup extent tree, skipping it Couldn't setup
>> device tree Superblock thinks the generation is 120260 Superblock
>> thinks the level is 1 Well block 7939758882816(gen: 120264 level: 1)
>> seems good, but generation/level doesn't match, want gen: 120260
>> level: 1 Well block 7939747938304(gen: 120263 level: 1) seems good,
>> but generation/level doesn't match, want gen: 120260 level: 1 Well
>> block 7939756146688(gen: 120262 level: 1) seems good, but
>> generation/level doesn't match, want gen: 120260 level: 1 Well block
>> 7939751559168(gen: 120261 level: 0) seems good, but generation/level
>> doesn't match, want gen: 120260 level: 1
>>
>> *** A large selection of block references was removed due to
>> character count... if needed, I can resend with the full output.
>>
>> Well block 1316967743488(gen: 1293 level: 0) seems good, but
>> generation/level doesn't match, want gen: 120260 level: 1 Well block
>> 1316909662208(gen: 1283 level: 0) seems good, but generation/level
>> doesn't match, want gen: 120260 level: 1 Well block 1316908711936(gen:
>> 1283 level: 0) seems good, but generation/level doesn't match, want
>> gen: 120260 level: 1 root@onyx:/home#
>>
>> Any help or assistance would be greatly appreciated. Important data
>> has been backed up, however if it's possible to recover without
>> thrashing the entire volume, that would be preferred.
>
> First thing first, don't expect too much about magically turning the fs back to fully functional status.
> Transid error is always tricky for btrfs.
>
>
> But for your case, I'm guessing your sdb1 does not have the latest super block.
> We have newer tree roots on disk, but older super block.
>
> Maybe you would like to try "btrfs ins dump-tree" on all the involved disks, and find if there is newer super blocks.
>
> Thanks,
> Qu
>>
>> Regards,
>> Weldon
>>
>
[-- Attachment #2: btrfs-find-root.txt --]
[-- Type: text/plain, Size: 67032 bytes --]
root@onyx:/home/home# btrfs-find-root /dev/sdb1
parent transid verify failed on 7939752886272 wanted 120260 found 120262
parent transid verify failed on 7939752886272 wanted 120260 found 120265
parent transid verify failed on 7939752886272 wanted 120260 found 120265
Ignoring transid failure
WARNING: could not setup extent tree, skipping it
Couldn't setup device tree
Superblock thinks the generation is 120260
Superblock thinks the level is 1
Well block 7939758882816(gen: 120264 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939747938304(gen: 120263 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939756146688(gen: 120262 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939751559168(gen: 120261 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939751313408(gen: 120261 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939743727616(gen: 120261 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939742810112(gen: 120261 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939766599680(gen: 120260 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939753590784(gen: 120260 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939753263104(gen: 120260 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939759112192(gen: 120259 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939747135488(gen: 120259 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939743252480(gen: 120259 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939742842880(gen: 120259 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939737108480(gen: 120259 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939737059328(gen: 120259 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939752476672(gen: 120258 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939753115648(gen: 120257 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939753082880(gen: 120257 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939740827648(gen: 120257 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939735781376(gen: 120257 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939762601984(gen: 120256 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939758243840(gen: 120256 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939757883392(gen: 120256 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939744022528(gen: 120256 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939745251328(gen: 120253 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939745153024(gen: 120253 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939651698688(gen: 120252 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939636117504(gen: 120251 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939650650112(gen: 120250 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939650519040(gen: 120250 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939650469888(gen: 120250 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939650338816(gen: 120250 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939648258048(gen: 120250 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939647012864(gen: 120250 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939646996480(gen: 120250 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939646570496(gen: 120250 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939646554112(gen: 120250 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939646341120(gen: 120250 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939637297152(gen: 120250 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939635953664(gen: 120250 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939635331072(gen: 120250 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939635183616(gen: 120250 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939634905088(gen: 120250 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939643883520(gen: 120249 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939636445184(gen: 120249 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939634954240(gen: 120248 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939627991040(gen: 120248 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939627892736(gen: 120248 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939637133312(gen: 120247 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939637100544(gen: 120247 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939628531712(gen: 120247 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939627827200(gen: 120247 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939627368448(gen: 120247 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939626401792(gen: 120247 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939615784960(gen: 120245 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939608838144(gen: 120243 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939597877248(gen: 120241 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939620323328(gen: 120240 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939606282240(gen: 120240 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939605004288(gen: 120240 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939609886720(gen: 120238 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939599286272(gen: 120238 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939598958592(gen: 120238 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939591323648(gen: 120238 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939590651904(gen: 120238 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939600449536(gen: 120236 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939585277952(gen: 120236 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939581067264(gen: 120236 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939596124160(gen: 120235 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939586818048(gen: 120235 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939584409600(gen: 120232 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939579052032(gen: 120232 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939578019840(gen: 120232 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939573317632(gen: 120232 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939573301248(gen: 120232 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939573268480(gen: 120232 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939567976448(gen: 120232 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939567828992(gen: 120232 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939562831872(gen: 120232 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939549085696(gen: 120231 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939522396160(gen: 120230 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939509747712(gen: 120229 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939536273408(gen: 120228 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939535732736(gen: 120228 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939535650816(gen: 120228 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939535388672(gen: 120228 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939535355904(gen: 120228 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939535011840(gen: 120228 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939526557696(gen: 120228 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939525918720(gen: 120228 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939525705728(gen: 120228 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939525672960(gen: 120228 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939522543616(gen: 120228 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939515154432(gen: 120228 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939514990592(gen: 120228 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939499229184(gen: 120227 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939521822720(gen: 120226 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939510026240(gen: 120226 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939494445056(gen: 120224 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939493986304(gen: 120224 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939493888000(gen: 120224 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939492085760(gen: 120224 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939491643392(gen: 120224 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939491495936(gen: 120224 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939491463168(gen: 120224 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939481583616(gen: 120222 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939485122560(gen: 120221 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939501948928(gen: 120220 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939479552000(gen: 120220 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939478945792(gen: 120219 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939478913024(gen: 120219 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939478798336(gen: 120219 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939473178624(gen: 120219 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939488972800(gen: 120218 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939478224896(gen: 120218 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939483631616(gen: 120217 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939483598848(gen: 120217 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939467755520(gen: 120217 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939467444224(gen: 120217 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939497099264(gen: 120216 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939477553152(gen: 120216 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939477520384(gen: 120216 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939477504000(gen: 120216 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939468017664(gen: 120216 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939441459200(gen: 120213 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939430563840(gen: 120212 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939448012800(gen: 120211 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939446898688(gen: 120211 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939436724224(gen: 120211 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939436134400(gen: 120211 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939436101632(gen: 120211 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939434135552(gen: 120210 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939434119168(gen: 120210 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939431497728(gen: 120210 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939431383040(gen: 120210 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939427647488(gen: 120210 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939427614720(gen: 120210 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939426648064(gen: 120210 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939426631680(gen: 120210 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939387047936(gen: 120209 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939366551552(gen: 120208 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939354443776(gen: 120207 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939341828096(gen: 120205 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939336142848(gen: 120203 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939371646976(gen: 120200 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939358015488(gen: 120200 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939357933568(gen: 120200 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939347496960(gen: 120200 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939334930432(gen: 120200 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939313893376(gen: 120199 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939357851648(gen: 120198 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939331325952(gen: 120198 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939331293184(gen: 120198 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939331227648(gen: 120198 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939306700800(gen: 120197 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939330539520(gen: 120196 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939330523136(gen: 120196 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939300081664(gen: 120196 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939299082240(gen: 120196 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939305291776(gen: 120195 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5890568962048(gen: 120194 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5890235514880(gen: 120193 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5890131230720(gen: 120192 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5890045689856(gen: 120191 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889889697792(gen: 120190 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889733132288(gen: 120189 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889611497472(gen: 120188 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5158251642880(gen: 120187 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5158115934208(gen: 120186 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5158033784832(gen: 120185 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5157908774912(gen: 120184 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5157731860480(gen: 120183 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5157615616000(gen: 120182 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5157479743488(gen: 120181 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5157214978048(gen: 120180 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 4145641013248(gen: 120179 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 4145580507136(gen: 120178 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 4145411260416(gen: 120177 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 4145035182080(gen: 120176 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 4144753639424(gen: 120175 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993621860352(gen: 120174 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993373298688(gen: 120173 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993313824768(gen: 120172 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2992575561728(gen: 120171 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2253548748800(gen: 120170 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2253048528896(gen: 120169 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2252917866496(gen: 120168 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1317371707392(gen: 120167 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1317010079744(gen: 120166 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316792532992(gen: 120165 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316574920704(gen: 120164 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 426108878848(gen: 120163 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 425887531008(gen: 120162 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 425637904384(gen: 120161 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 986431488(gen: 120160 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 437010432(gen: 120159 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 245055488(gen: 120158 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 151650304(gen: 120157 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940334354432(gen: 120156 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940320706560(gen: 120155 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940345430016(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940345282560(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940345167872(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940345102336(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940345053184(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940345020416(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940342513664(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940342366208(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940342054912(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940335157248(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940331634688(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940331585536(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940331536384(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940331438080(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940327014400(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940326080512(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940325982208(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940325785600(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940325703680(gen: 120154 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940316561408(gen: 120153 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940331470848(gen: 120152 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940318347264(gen: 120152 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940317151232(gen: 120152 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940321902592(gen: 120150 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940321869824(gen: 120150 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940310843392(gen: 120150 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940310581248(gen: 120150 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940310433792(gen: 120150 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940300275712(gen: 120149 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940290904064(gen: 120148 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940316987392(gen: 120147 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940316790784(gen: 120147 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940304486400(gen: 120147 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940292542464(gen: 120147 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940290985984(gen: 120147 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940290707456(gen: 120147 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940279468032(gen: 120146 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940284841984(gen: 120145 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940299440128(gen: 120144 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940288757760(gen: 120144 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940281466880(gen: 120143 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940276256768(gen: 120143 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940276142080(gen: 120143 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940275847168(gen: 120143 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940275830784(gen: 120143 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940275273728(gen: 120143 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940262641664(gen: 120142 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940256858112(gen: 120141 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940246290432(gen: 120140 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940239523840(gen: 120138 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940248829952(gen: 120136 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940248748032(gen: 120136 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940244324352(gen: 120136 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940235362304(gen: 120136 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940235280384(gen: 120136 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940234772480(gen: 120136 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940199006208(gen: 120135 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940177215488(gen: 120134 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940175560704(gen: 120133 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940163321856(gen: 120132 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940199366656(gen: 120131 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940195057664(gen: 120131 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940185620480(gen: 120131 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940173201408(gen: 120131 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940172627968(gen: 120131 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940156669952(gen: 120130 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940165877760(gen: 120129 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940135485440(gen: 120126 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940145823744(gen: 120125 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940121378816(gen: 120124 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940158816256(gen: 120123 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940128522240(gen: 120123 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940112564224(gen: 120122 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940107206656(gen: 120121 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940098326528(gen: 120120 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940113465344(gen: 120119 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940104175616(gen: 120119 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940102094848(gen: 120119 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940090036224(gen: 120118 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940056776704(gen: 120117 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940051763200(gen: 120116 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940077436928(gen: 120115 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940077420544(gen: 120115 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940047814656(gen: 120115 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940047077376(gen: 120115 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940046815232(gen: 120115 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940037885952(gen: 120114 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940031823872(gen: 120113 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940023173120(gen: 120112 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940016340992(gen: 120111 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940013916160(gen: 120110 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940011130880(gen: 120109 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940003364864(gen: 120108 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940000268288(gen: 120107 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939993583616(gen: 120106 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939989831680(gen: 120105 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940007346176(gen: 120104 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940002185216(gen: 120104 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940001955840(gen: 120104 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939995615232(gen: 120104 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939995140096(gen: 120104 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939991289856(gen: 120104 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939975872512(gen: 120102 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939968958464(gen: 120101 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939976904704(gen: 120100 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939962388480(gen: 120099 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939953999872(gen: 120098 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939918430208(gen: 120097 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939899047936(gen: 120096 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939944857600(gen: 120095 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939913302016(gen: 120095 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939910926336(gen: 120095 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939898032128(gen: 120095 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939939942400(gen: 120094 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939939909632(gen: 120094 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939939565568(gen: 120094 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939939024896(gen: 120094 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939938926592(gen: 120094 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939931914240(gen: 120094 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939885662208(gen: 120094 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939868426240(gen: 120092 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939914956800(gen: 120091 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939880927232(gen: 120091 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939873439744(gen: 120091 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939873390592(gen: 120091 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939873357824(gen: 120091 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939864739840(gen: 120090 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939854336000(gen: 120089 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939851124736(gen: 120088 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939846995968(gen: 120087 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939841277952(gen: 120086 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939837837312(gen: 120085 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939833872384(gen: 120084 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939830661120(gen: 120083 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939836116992(gen: 120082 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939833446400(gen: 120082 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939832479744(gen: 120082 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939822288896(gen: 120081 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939815915520(gen: 120080 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939809148928(gen: 120079 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939804758016(gen: 120078 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939804332032(gen: 120077 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939798515712(gen: 120076 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939796647936(gen: 120075 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939792568320(gen: 120074 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939784605696(gen: 120072 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939795255296(gen: 120071 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939795189760(gen: 120071 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939795058688(gen: 120071 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939789553664(gen: 120071 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939786326016(gen: 120071 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939785981952(gen: 120071 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939785834496(gen: 120071 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939785736192(gen: 120071 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939781918720(gen: 120070 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939783000064(gen: 120069 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939780591616(gen: 120069 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939780280320(gen: 120069 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939787784192(gen: 120068 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939787669504(gen: 120068 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939787227136(gen: 120068 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939776200704(gen: 120066 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939781427200(gen: 120065 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939777740800(gen: 120064 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939777658880(gen: 120064 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939616882688(gen: 120044 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889547059200(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889546567680(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889545453568(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889545437184(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889545109504(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889545076736(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889544994816(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889544945664(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889544601600(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889544552448(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889538473984(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889532526592(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889532510208(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889532461056(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889531871232(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889518059520(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889517715456(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889515487232(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889513144320(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889513127936(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889513111552(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889513046016(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889513013248(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889511276544(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889511243776(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889511227392(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889511112704(gen: 119996 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993294147584(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993293918208(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993293705216(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993293688832(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993293574144(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993291640832(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993290133504(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993280958464(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993280401408(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993280352256(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993280253952(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993280024576(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993148903424(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993148805120(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993148739584(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993148641280(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993148608512(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993148329984(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993148313600(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993148297216(gen: 119985 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 425387589632(gen: 119977 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 425387556864(gen: 119977 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 451919872(gen: 119977 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940318658560(gen: 119973 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940323475456(gen: 119971 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940293574656(gen: 119971 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940010016768(gen: 119930 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939871834112(gen: 119916 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939857809408(gen: 119916 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2253048512512(gen: 119877 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939972005888(gen: 119867 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 425753673728(gen: 119848 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 255868928(gen: 119848 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 217202688(gen: 119848 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940242685952(gen: 119846 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940166057984(gen: 119845 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940161470464(gen: 119844 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940007985152(gen: 119844 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940007804928(gen: 119844 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940007772160(gen: 119844 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939775709184(gen: 119837 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939466985472(gen: 119833 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939397877760(gen: 119833 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939388948480(gen: 119833 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5890385969152(gen: 119832 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2252848005120(gen: 119776 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 426169729024(gen: 119776 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 548470784(gen: 119775 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 548143104(gen: 119775 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 481902592(gen: 119774 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940261232640(gen: 119773 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940261216256(gen: 119773 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940188094464(gen: 119772 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940108271616(gen: 119771 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939984506880(gen: 119769 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939905454080(gen: 119768 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939833790464(gen: 119767 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939601547264(gen: 119763 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939601481728(gen: 119763 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5158076284928(gen: 119757 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 426183770112(gen: 119728 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 4144817799168(gen: 119640 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 4144679124992(gen: 119618 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316688363520(gen: 119615 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 425488039936(gen: 119615 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940273618944(gen: 119613 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940107698176(gen: 119611 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939295526912(gen: 119601 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889504755712(gen: 119454 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889504722944(gen: 119454 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889504706560(gen: 119454 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889504690176(gen: 119454 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939973693440(gen: 119199 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939784949760(gen: 119189 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2252827738112(gen: 118426 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316571201536(gen: 118416 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316571168768(gen: 118416 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939385901056(gen: 117763 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939372318720(gen: 117762 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2992562585600(gen: 117602 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316836196352(gen: 117592 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316568612864(gen: 117589 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 509247488(gen: 117579 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940324458496(gen: 117559 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940308287488(gen: 117550 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940313268224(gen: 117549 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939943399424(gen: 117396 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939753607168(gen: 117338 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939742793728(gen: 117338 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939742728192(gen: 117338 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939745873920(gen: 117337 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939736158208(gen: 117333 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939626696704(gen: 117318 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939507339264(gen: 117288 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316540940288(gen: 116911 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316543922176(gen: 116909 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316541005824(gen: 116909 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316524425216(gen: 115428 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993467457536(gen: 114742 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316494049280(gen: 114712 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316439228416(gen: 114712 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316438949888(gen: 114712 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316438786048(gen: 114712 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316438753280(gen: 114712 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316438720512(gen: 114712 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316438376448(gen: 114712 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316437934080(gen: 114712 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 66158592(gen: 114686 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940237180928(gen: 114646 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940226564096(gen: 114640 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2252784746496(gen: 113899 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 54804480(gen: 113364 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 54755328(gen: 113364 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 54722560(gen: 113364 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 54444032(gen: 113364 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 54329344(gen: 113364 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 54280192(gen: 113364 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993308467200(gen: 112701 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2993320132608(gen: 112700 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 53886976(gen: 112627 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 53837824(gen: 112627 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 53084160(gen: 111640 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 52969472(gen: 111640 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 52789248(gen: 111640 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 52297728(gen: 111640 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 52019200(gen: 111640 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 51920896(gen: 111640 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 51183616(gen: 110835 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 46383104(gen: 110276 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7940237393920(gen: 109340 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 33406976(gen: 108273 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 33292288(gen: 108273 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 33275904(gen: 108273 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2252747472896(gen: 107194 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939818553344(gen: 106707 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5890047852544(gen: 106212 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5890047541248(gen: 106212 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5890048245760(gen: 106211 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5890043150336(gen: 106210 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5158254903296(gen: 106198 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 2252747145216(gen: 103991 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 7939838279680(gen: 96586 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 252772352(gen: 88434 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 31260672(gen: 83964 level: 1) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 393936896(gen: 63693 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889607598080(gen: 46873 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889562411008(gen: 46857 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5889532641280(gen: 46846 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 5158067650560(gen: 31731 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 4145498783744(gen: 10533 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316967743488(gen: 1293 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316909662208(gen: 1283 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
Well block 1316908711936(gen: 1283 level: 0) seems good, but generation/level doesn't match, want gen: 120260 level: 1
root@onyx:/home/home#
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: BTRFS fails mount after power failure
2021-08-24 0:52 ` weldon
@ 2021-08-24 1:54 ` Qu Wenruo
0 siblings, 0 replies; 6+ messages in thread
From: Qu Wenruo @ 2021-08-24 1:54 UTC (permalink / raw)
To: weldon, linux-btrfs
On 2021/8/24 上午8:52, weldon@newfietech.com wrote:
> I've attached the complete results of the btrfs-find-root command.
>
> If I understood your directions correctly, below is the result:
>
> root@onyx:/home# btrfs check -r 7939758882816 /dev/sdb1
> Opening filesystem to check...
> parent transid verify failed on 7939758882816 wanted 120260 found 120264
> parent transid verify failed on 7939758882816 wanted 120260 found 120264
> parent transid verify failed on 7939758882816 wanted 120260 found 120264
> Ignoring transid failure
> parent transid verify failed on 7939751723008 wanted 120264 found 120262
> parent transid verify failed on 7939751723008 wanted 120264 found 120266
> parent transid verify failed on 7939751723008 wanted 120264 found 120266
> Ignoring transid failure
> parent transid verify failed on 7939735683072 wanted 120264 found 120263
> parent transid verify failed on 7939735683072 wanted 120261 found 120264
> Ignoring transid failure
> parent transid verify failed on 7939734437888 wanted 120264 found 120253
Well, there are more transid errors than I thought.
This mostly means the RAID controller is not resilvering the correct
data out.
Thus metadata COW is broken.
Thus the only thing we can do is data salvaging using btrfs-restore,
which is far from ideal.
Thus it's recommended to restore from backup.
In this particular case, since the RAID5 controller is involved, there
isn't much thing we can do from btrfs realm.
Thus I'd recommend to not to use the RAID5 controller, but connect all
those disks directly so the kernel can manage them directly.
Then build btrfs RAID10 on top of these disks.
The cost is obvious, only half of the disks can really be utilized, but
it should get rid of the unreliable RAID5 controller.
Thanks,
Qu
> Checking filesystem on /dev/sdb1
> UUID: 7f500ee1-32b7-45a3-b1e9-deb7e1f59632
> [1/7] checking root items
> Error: could not find extent items for root 18446744073709551607
> ERROR: failed to repair root items: No such file or directory
> root@onyx:/home#
>
> root@onyx:/home# btrfs check -r 7939747938304 /dev/sdb1
> Opening filesystem to check...
> parent transid verify failed on 7939747938304 wanted 120260 found 120263
> parent transid verify failed on 7939747938304 wanted 120260 found 120265
> parent transid verify failed on 7939747938304 wanted 120260 found 120265
> Ignoring transid failure
> ERROR: could not setup extent tree
> ERROR: cannot open file system
> root@onyx:/home#
>
> root@onyx:/home# btrfs check -r 7939756146688 /dev/sdb1
> Opening filesystem to check...
> parent transid verify failed on 7939756146688 wanted 120260 found 120262
> parent transid verify failed on 7939756146688 wanted 120260 found 120264
> parent transid verify failed on 7939756146688 wanted 120260 found 120264
> Ignoring transid failure
> ERROR: could not setup extent tree
> ERROR: cannot open file system
> root@onyx:/home#
>
> root@onyx:/home# btrfs check -r 7939751559168 /dev/sdb1
> Opening filesystem to check...
> parent transid verify failed on 7939751559168 wanted 120260 found 120261
> parent transid verify failed on 7939751559168 wanted 120260 found 120261
> parent transid verify failed on 7939751559168 wanted 120260 found 120261
> Ignoring transid failure
> ERROR: could not setup extent tree
> ERROR: cannot open file system
> root@onyx:/home#
>
> Thanks again,
> Weldon
>
>
> -----Original Message-----
> From: Qu Wenruo <quwenruo.btrfs@gmx.com>
> Sent: August 23, 2021 6:39 PM
> To: weldon@newfietech.com; linux-btrfs@vger.kernel.org
> Subject: Re: BTRFS fails mount after power failure
>
>
>
> On 2021/8/24 上午8:10, weldon@newfietech.com wrote:
>> Thank you for the reply Qu.
>>
>> The hardware setup is a bit wonky in a home lab, but is as follows:
>>
>> Dell PowerEdge R510 Chassis
>> Dell PERC H700
>> 6 * 4TB SATA Disks in a RAID 5 configuration
>
> The RAID5 is not provided by btrfs, but some hardware RAID controller?
>
> Then we don't need to bother the btrfs RAID5 bug.
>
> But still, this means the RAID controller or the hdd is not doing proper flush/fua.
>
> This means, next time your UPS went down or a kernel crash happens, you may still hit a similar problem.
>
> And this time, we're pretty sure it's less possible to blame btrfs code.
>
>
>> ESXi 6.5 hypervisor sees storage as local DELL Disk, 18.19TB
>>
>> 17.66TB Provisioned as a Datastore on the hypervisor, VMFS5.
>> - 14.5TB provisioned as a vmdk and presented as local disk to Ubuntu
>> virtual machine, mounted as /data (btrfs)
>> - 200GB provisioned as vmdk and presented as local disk to Ubuntu
>> virtual machine, mounted as / (ext4)
>>
>> Happy and willing to try any suggestions you may have.
>>
>> root@onyx:/home# btrfs ins dump-tree /dev/sdb1
>
> My bad, I mean "btrfs ins dump-super -fFa", but that's for the case of btrfs RAID5 setup.
>
> Since you're using hardware RAID5 controller, we can go direct to the recovery part.
>
> Your previous find-root output would be pretty helpful.
>
> You can try btrfs-check with -r option:
>
> # btrfs check -r 7939758882816 /dev/sdb1
>
> To see how many errors it throws. if it had almost no error, then it has a pretty high chance to recover the data.
>
> You can also try other bytenr from your find-root output, but I guess you only need to try the first 4 bytenrs.
>
> Thanks,
> Qu
>
>> btrfs-progs v5.4.1
>> parent transid verify failed on 7939752886272 wanted 120260 found
>> 120262 parent transid verify failed on 7939752886272 wanted 120260
>> found 120265 parent transid verify failed on 7939752886272 wanted
>> 120260 found 120265 Ignoring transid failure
>> WARNING: could not setup extent tree, skipping it Couldn't setup
>> device tree
>> ERROR: unable to open /dev/sdb1
>> root@onyx:/home#
>>
>>
>> Thanks in advance,
>> Weldon
>>
>>
>> -----Original Message-----
>> From: Qu Wenruo <quwenruo.btrfs@gmx.com>
>> Sent: August 23, 2021 5:55 PM
>> To: weldon@newfietech.com; linux-btrfs@vger.kernel.org
>> Subject: Re: BTRFS fails mount after power failure
>>
>>
>>
>> On 2021/8/24 上午4:52, weldon@newfietech.com wrote:
>>> Good day folks,
>>>
>>> I awoke this morning to find that my UPS had died overnight and my
>>> Ubuntu server with a 14.5TB (Raid 5) BTRFS volume went down with it.
>>
>> RAID5 has known write hole bug, and although that bug won't cause immediate problems, it slowly degrades the whole array with each corrupted sector or unexpected power loss.
>>
>> This would eventually bring down the array with enough degradation.
>>
>>> The machine
>>> rebooted fine and the hardware reports no errors, however the BTRFS
>>> volume will no longer mount. The OS boots fine, the 14.5TB volume
>>> is for data storage only. gparted shows the volume/partition, and
>>> correctly reports space used as well as total size. I've never
>>> encountered this type of issue over the past year while using btrfs
>>> and I'm not sure where to start. A number of google search results
>>> express caution when attempting to recover/repair, so I'm hoping for some expert advice.
>>>
>>> My dmesg log exceeds the 100,000 bytes restriction, so I'm unable to
>>> attach it, so please ask if there's anything specific I can include otherwise.
>>>
>>> # uname -a
>>> Linux onyx 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC
>>> 2021
>>> x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> # btrfs --version
>>> btrfs-progs v5.4.1
>>>
>>> # btrfs fi show
>>> Label: 'Data' uuid: 7f500ee1-32b7-45a3-b1e9-deb7e1f59632
>>> Total devices 1 FS bytes used 7.17TiB
>>> devid 1 size 14.50TiB used 7.40TiB path /dev/sdb1
>>>
>>> # dmesg | grep sdb
>>> [ 2.312875] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
>>> CAPACITY(16).
>>> [ 2.313010] sd 32:0:1:0: [sdb] 31138512896 512-byte logical blocks: (15.9
>>> TB/14.5 TiB)
>>> [ 2.313062] sd 32:0:1:0: [sdb] Write Protect is off
>>> [ 2.313065] sd 32:0:1:0: [sdb] Mode Sense: 61 00 00 00
>>> [ 2.313116] sd 32:0:1:0: [sdb] Cache data unavailable
>>> [ 2.313119] sd 32:0:1:0: [sdb] Assuming drive cache: write through
>>> [ 2.333321] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
>>> CAPACITY(16).
>>> [ 2.396761] sdb: sdb1
>>> [ 2.397170] sd 32:0:1:0: [sdb] Very big device. Trying to use READ
>>> CAPACITY(16).
>>> [ 2.397261] sd 32:0:1:0: [sdb] Attached SCSI disk
>>> [ 4.709963] BTRFS: device label Data devid 1 transid 120260 /dev/sdb1
>>> [ 21.849570] BTRFS info (device sdb1): disk space caching is enabled
>>> [ 21.849573] BTRFS info (device sdb1): has skinny extents
>>> [ 22.023224] BTRFS error (device sdb1): parent transid verify failed on
>>> 7939752886272 wanted 120260 found 120262
>>> [ 22.047940] BTRFS error (device sdb1): parent transid verify failed on
>>> 7939752886272 wanted 120260 found 120265
>>
>> This already shows some mismatch in on-disk data and recovered data from parity.
>>
>> This shows the on-disk data and parity have drifted from each other, exactly the write hole problem.
>>
>> Furthermore, the disk has newer data than what we expect.
>>
>> What's the device model? It looks like a misbehavior, not sure if it's from the hardware, or the btrfs code.
>> As RAID56 is already marked as unsafe for a while, not that much love nor code fix is directed to RAID56, thus both cases are possible.
>>
>>> [ 22.047949] BTRFS warning (device sdb1): failed to read tree root
>>> [ 22.089003] BTRFS error (device sdb1): open_ctree failed
>>>
>>> root@onyx:/home/weldon# btrfs-find-root /dev/sdb1 parent transid
>>> verify failed on 7939752886272 wanted 120260 found 120262 parent
>>> transid verify failed on 7939752886272 wanted 120260 found 120265
>>> parent transid verify failed on 7939752886272 wanted 120260 found
>>> 120265 Ignoring transid failure
>>> WARNING: could not setup extent tree, skipping it Couldn't setup
>>> device tree Superblock thinks the generation is 120260 Superblock
>>> thinks the level is 1 Well block 7939758882816(gen: 120264 level: 1)
>>> seems good, but generation/level doesn't match, want gen: 120260
>>> level: 1 Well block 7939747938304(gen: 120263 level: 1) seems good,
>>> but generation/level doesn't match, want gen: 120260 level: 1 Well
>>> block 7939756146688(gen: 120262 level: 1) seems good, but
>>> generation/level doesn't match, want gen: 120260 level: 1 Well block
>>> 7939751559168(gen: 120261 level: 0) seems good, but generation/level
>>> doesn't match, want gen: 120260 level: 1
>>>
>>> *** A large selection of block references was removed due to
>>> character count... if needed, I can resend with the full output.
>>>
>>> Well block 1316967743488(gen: 1293 level: 0) seems good, but
>>> generation/level doesn't match, want gen: 120260 level: 1 Well block
>>> 1316909662208(gen: 1283 level: 0) seems good, but generation/level
>>> doesn't match, want gen: 120260 level: 1 Well block 1316908711936(gen:
>>> 1283 level: 0) seems good, but generation/level doesn't match, want
>>> gen: 120260 level: 1 root@onyx:/home#
>>>
>>> Any help or assistance would be greatly appreciated. Important data
>>> has been backed up, however if it's possible to recover without
>>> thrashing the entire volume, that would be preferred.
>>
>> First thing first, don't expect too much about magically turning the fs back to fully functional status.
>> Transid error is always tricky for btrfs.
>>
>>
>> But for your case, I'm guessing your sdb1 does not have the latest super block.
>> We have newer tree roots on disk, but older super block.
>>
>> Maybe you would like to try "btrfs ins dump-tree" on all the involved disks, and find if there is newer super blocks.
>>
>> Thanks,
>> Qu
>>>
>>> Regards,
>>> Weldon
>>>
>>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-08-24 1:59 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-23 20:52 BTRFS fails mount after power failure weldon
2021-08-23 23:54 ` Qu Wenruo
2021-08-24 0:10 ` weldon
2021-08-24 0:38 ` Qu Wenruo
2021-08-24 0:52 ` weldon
2021-08-24 1:54 ` Qu Wenruo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).