All files are damaged after btrfs restore

All files are damaged after btrfs restore

[Sebastian Roller]

Hello all.
Sorry for asking here directly, but I'm in a desperate situation and
out of options.
I have a 72 TB btrfs filesystem which functions as a backup drive.
After a recent controller hardware failure while the backup was
running, both original and backup fs were severely damaged.

Kernel version is 5.7.7. btrfs-progs is (now) 5.9.

At the moment I am unable to mount the btrfs filesystem.

root@hikitty:~$ mount -t btrfs -o ro,recovery /dev/sdf1 /mnt/
mount: wrong fs type, bad option, bad superblock on /dev/sdf1,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

[165097.777496] BTRFS warning (device sdf1): 'recovery' is deprecated,
use 'usebackuproot' instead
[165097.777500] BTRFS info (device sdf1): trying to use backup root at
mount time
[165097.777502] BTRFS info (device sdf1): disk space caching is enabled
[165097.777505] BTRFS info (device sdf1): has skinny extents
[165101.721250] BTRFS error (device sdf1): bad tree block start, want
126718415241216 have 0
[165101.750951] BTRFS error (device sdf1): bad tree block start, want
126718415241216 have 0
[165101.755753] BTRFS error (device sdf1): failed to verify dev
extents against chunks: -5
[165101.895065] BTRFS error (device sdf1): open_ctree failed


Since I desperately need the data I ran btrfs restore.
root@hikitty:~$ install/btrfs-progs-5.9/btrfs -v restore -i -s -m -S
--path-regex '^/(|@(|/backup(|/home(|/.*))))$' /dev/sdf1
/mnt/dumpo/home/
checksum verify failed on 109911545430016 found 000000B6 wanted 00000000
checksum verify failed on 109911545462784 found 000000B6 wanted 00000000
checksum verify failed on 57767345897472 found 000000B6 wanted 00000000
Restoring /mnt/dumpo/home/@
Restoring /mnt/dumpo/home/@/backup
Restoring /mnt/dumpo/home/@/backup/home
…
(2.1 GB of log file)
…
Done searching /@/backup/home
Reached the end of the tree searching the directory
Reached the end of the tree searching the directory
Reached the end of the tree searching the directory


Using that restore I was able to restore approx. 7 TB of the
originally stored 22 TB under that directory.
Unfortunately nearly all the files are damaged. Small text files are
still OK. But every larger binary file is useless.
Is there any possibility to fix the filesystem in a way, that I get
the data less damaged?

So far I ran no btrfs check --repair.

Since the original and the backup have been damaged any help would be
highly appreciated.
Thanks for your assistance.

Kind regards,
Sebastian Roller

----------------  Attachment. All outputs. -------------------
uname -a
Linux hikitty 5.7.7-1.el7.elrepo.x86_64 #1 SMP Wed Jul 1 11:53:16 EDT
2020 x86_64 x86_64 x86_64 GNU/Linux


root@hikitty:~$ install/btrfs-progs-5.9/btrfs --version
btrfs-progs v5.9
(Version v5.10 fails to compile)


root@hikitty:~$ btrfs fi show
Label: 'history'  uuid: 56051c5f-fca6-4d54-a04e-1c1d8129fe56
        Total devices 1 FS bytes used 68.37TiB
        devid    2 size 72.77TiB used 68.59TiB path /dev/sdf1


root@hikitty:~$ mount -t btrfs -o ro,recovery /dev/sdf1 /mnt/hist/
mount: wrong fs type, bad option, bad superblock on /dev/sdf1,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

[165097.777496] BTRFS warning (device sdf1): 'recovery' is deprecated,
use 'usebackuproot' instead
[165097.777500] BTRFS info (device sdf1): trying to use backup root at
mount time
[165097.777502] BTRFS info (device sdf1): disk space caching is enabled
[165097.777505] BTRFS info (device sdf1): has skinny extents
[165101.721250] BTRFS error (device sdf1): bad tree block start, want
126718415241216 have 0
[165101.750951] BTRFS error (device sdf1): bad tree block start, want
126718415241216 have 0
[165101.755753] BTRFS error (device sdf1): failed to verify dev
extents against chunks: -5
[165101.895065] BTRFS error (device sdf1): open_ctree failed


root@hikitty:~$ btrfs rescue super-recover -v /dev/sdf1
All Devices:
        Device: id = 2, name = /dev/sdh1

Before Recovering:
        [All good supers]:
                device name = /dev/sdh1
                superblock bytenr = 65536

                device name = /dev/sdh1
                superblock bytenr = 67108864

                device name = /dev/sdh1
                superblock bytenr = 274877906944

        [All bad supers]:

All supers are valid, no need to recover


root@hikitty:/mnt$ btrfs rescue chunk-recover /dev/sdf1
Scanning: DONE in dev0
checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
bytenr mismatch, want=124762809384960, have=0
open with broken chunk error
Chunk tree recovery failed

^^ This was btrfs v4.14


root@hikitty:~$ install/btrfs-progs-5.9/btrfs check --readonly /dev/sdi1
Opening filesystem to check...
checksum verify failed on 99593231630336 found 000000B6 wanted 00000000
checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
bad tree block 124762809384960, bytenr mismatch, want=124762809384960, have=0
ERROR: failed to read block groups: Input/output error
ERROR: cannot open file system


FIRST MOUNT AT BOOT TIME AFTER DESASTER
Feb 15 08:05:11 hikitty kernel: BTRFS info (device sdf1): disk space
caching is enabled
Feb 15 08:05:11 hikitty kernel: BTRFS info (device sdf1): has skinny extents
Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
block start, want 141944039161856 have 0
Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
corrected: ino 0 off 141944039161856 (dev /dev/sdf1 sector 3974114336)
Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
corrected: ino 0 off 141944039165952 (dev /dev/sdf1 sector 3974114344)
Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
corrected: ino 0 off 141944039170048 (dev /dev/sdf1 sector 3974114352)
Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
corrected: ino 0 off 141944039174144 (dev /dev/sdf1 sector 3974114360)
Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
block start, want 141944037851136 have 0
Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
corrected: ino 0 off 141944037851136 (dev /dev/sdf1 sector 3974111776)
Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
corrected: ino 0 off 141944037855232 (dev /dev/sdf1 sector 3974111784)
Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
corrected: ino 0 off 141944037859328 (dev /dev/sdf1 sector 3974111792)
Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
corrected: ino 0 off 141944037863424 (dev /dev/sdf1 sector 3974111800)
Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
block start, want 141944040767488 have 0
Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
corrected: ino 0 off 141944040767488 (dev /dev/sdf1 sector 3974117472)
Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
corrected: ino 0 off 141944040771584 (dev /dev/sdf1 sector 3974117480)
Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
block start, want 141944035147776 have 0
Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
block start, want 141944035115008 have 0
Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
block start, want 141944035131392 have 0
Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
block start, want 141944036327424 have 0
Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
block start, want 141944036278272 have 0
Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
block start, want 141944035164160 have 0
Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
block start, want 141944036294656 have 0
Feb 15 08:05:16 hikitty kernel: BTRFS error (device sdf1): failed to
verify dev extents against chunks: -5
Feb 15 08:05:16 hikitty kernel: BTRFS error (device sdf1): open_ctree failed

Re: All files are damaged after btrfs restore

[Chris Murphy]

On Tue, Feb 23, 2021 at 8:49 AM Sebastian Roller
<sebastian.roller@gmail.com> wrote:
>
> Hello all.
> Sorry for asking here directly, but I'm in a desperate situation and
> out of options.
> I have a 72 TB btrfs filesystem which functions as a backup drive.
> After a recent controller hardware failure while the backup was
> running, both original and backup fs were severely damaged.
>
> Kernel version is 5.7.7. btrfs-progs is (now) 5.9.
>
> At the moment I am unable to mount the btrfs filesystem.
>
> root@hikitty:~$ mount -t btrfs -o ro,recovery /dev/sdf1 /mnt/
> mount: wrong fs type, bad option, bad superblock on /dev/sdf1,
>        missing codepage or helper program, or other error
>
>        In some cases useful info is found in syslog - try
>        dmesg | tail or so.
>
> [165097.777496] BTRFS warning (device sdf1): 'recovery' is deprecated,
> use 'usebackuproot' instead
> [165097.777500] BTRFS info (device sdf1): trying to use backup root at
> mount time
> [165097.777502] BTRFS info (device sdf1): disk space caching is enabled
> [165097.777505] BTRFS info (device sdf1): has skinny extents
> [165101.721250] BTRFS error (device sdf1): bad tree block start, want
> 126718415241216 have 0
> [165101.750951] BTRFS error (device sdf1): bad tree block start, want
> 126718415241216 have 0
> [165101.755753] BTRFS error (device sdf1): failed to verify dev
> extents against chunks: -5
> [165101.895065] BTRFS error (device sdf1): open_ctree failed
>
>
> Since I desperately need the data I ran btrfs restore.
> root@hikitty:~$ install/btrfs-progs-5.9/btrfs -v restore -i -s -m -S
> --path-regex '^/(|@(|/backup(|/home(|/.*))))$' /dev/sdf1
> /mnt/dumpo/home/
> checksum verify failed on 109911545430016 found 000000B6 wanted 00000000
> checksum verify failed on 109911545462784 found 000000B6 wanted 00000000
> checksum verify failed on 57767345897472 found 000000B6 wanted 00000000
> Restoring /mnt/dumpo/home/@
> Restoring /mnt/dumpo/home/@/backup
> Restoring /mnt/dumpo/home/@/backup/home
> …
> (2.1 GB of log file)
> …
> Done searching /@/backup/home
> Reached the end of the tree searching the directory
> Reached the end of the tree searching the directory
> Reached the end of the tree searching the directory
>
>
> Using that restore I was able to restore approx. 7 TB of the
> originally stored 22 TB under that directory.
> Unfortunately nearly all the files are damaged. Small text files are
> still OK. But every larger binary file is useless.
> Is there any possibility to fix the filesystem in a way, that I get
> the data less damaged?
>
> So far I ran no btrfs check --repair.
>
> Since the original and the backup have been damaged any help would be
> highly appreciated.
> Thanks for your assistance.
>
> Kind regards,
> Sebastian Roller
>
> ----------------  Attachment. All outputs. -------------------
> uname -a
> Linux hikitty 5.7.7-1.el7.elrepo.x86_64 #1 SMP Wed Jul 1 11:53:16 EDT
> 2020 x86_64 x86_64 x86_64 GNU/Linux
>
>
> root@hikitty:~$ install/btrfs-progs-5.9/btrfs --version
> btrfs-progs v5.9
> (Version v5.10 fails to compile)
>
>
> root@hikitty:~$ btrfs fi show
> Label: 'history'  uuid: 56051c5f-fca6-4d54-a04e-1c1d8129fe56
>         Total devices 1 FS bytes used 68.37TiB
>         devid    2 size 72.77TiB used 68.59TiB path /dev/sdf1
>
>
> root@hikitty:~$ mount -t btrfs -o ro,recovery /dev/sdf1 /mnt/hist/
> mount: wrong fs type, bad option, bad superblock on /dev/sdf1,
>        missing codepage or helper program, or other error
>
>        In some cases useful info is found in syslog - try
>        dmesg | tail or so.
>
> [165097.777496] BTRFS warning (device sdf1): 'recovery' is deprecated,
> use 'usebackuproot' instead
> [165097.777500] BTRFS info (device sdf1): trying to use backup root at
> mount time
> [165097.777502] BTRFS info (device sdf1): disk space caching is enabled
> [165097.777505] BTRFS info (device sdf1): has skinny extents
> [165101.721250] BTRFS error (device sdf1): bad tree block start, want
> 126718415241216 have 0
> [165101.750951] BTRFS error (device sdf1): bad tree block start, want
> 126718415241216 have 0
> [165101.755753] BTRFS error (device sdf1): failed to verify dev
> extents against chunks: -5
> [165101.895065] BTRFS error (device sdf1): open_ctree failed
>
>
> root@hikitty:~$ btrfs rescue super-recover -v /dev/sdf1
> All Devices:
>         Device: id = 2, name = /dev/sdh1
>
> Before Recovering:
>         [All good supers]:
>                 device name = /dev/sdh1
>                 superblock bytenr = 65536
>
>                 device name = /dev/sdh1
>                 superblock bytenr = 67108864
>
>                 device name = /dev/sdh1
>                 superblock bytenr = 274877906944
>
>         [All bad supers]:
>
> All supers are valid, no need to recover
>
>
> root@hikitty:/mnt$ btrfs rescue chunk-recover /dev/sdf1
> Scanning: DONE in dev0
> checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
> checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
> checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> bytenr mismatch, want=124762809384960, have=0
> open with broken chunk error
> Chunk tree recovery failed
>
> ^^ This was btrfs v4.14
>
>
> root@hikitty:~$ install/btrfs-progs-5.9/btrfs check --readonly /dev/sdi1
> Opening filesystem to check...
> checksum verify failed on 99593231630336 found 000000B6 wanted 00000000
> checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> bad tree block 124762809384960, bytenr mismatch, want=124762809384960, have=0
> ERROR: failed to read block groups: Input/output error
> ERROR: cannot open file system
>
>
> FIRST MOUNT AT BOOT TIME AFTER DESASTER
> Feb 15 08:05:11 hikitty kernel: BTRFS info (device sdf1): disk space
> caching is enabled
> Feb 15 08:05:11 hikitty kernel: BTRFS info (device sdf1): has skinny extents
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944039161856 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944039161856 (dev /dev/sdf1 sector 3974114336)
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944039165952 (dev /dev/sdf1 sector 3974114344)
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944039170048 (dev /dev/sdf1 sector 3974114352)
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944039174144 (dev /dev/sdf1 sector 3974114360)
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944037851136 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944037851136 (dev /dev/sdf1 sector 3974111776)
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944037855232 (dev /dev/sdf1 sector 3974111784)
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944037859328 (dev /dev/sdf1 sector 3974111792)
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944037863424 (dev /dev/sdf1 sector 3974111800)
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944040767488 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944040767488 (dev /dev/sdf1 sector 3974117472)
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944040771584 (dev /dev/sdf1 sector 3974117480)
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944035147776 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944035115008 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944035131392 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944036327424 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944036278272 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944035164160 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944036294656 have 0
> Feb 15 08:05:16 hikitty kernel: BTRFS error (device sdf1): failed to
> verify dev extents against chunks: -5
> Feb 15 08:05:16 hikitty kernel: BTRFS error (device sdf1): open_ctree failed

I think you best chance is to start out trying to restore from a
recent snapshot. As long as the failed controller wasn't writing
totally spurious data in random locations, that snapshot should be
intact.

If there are no recent snapshots, and it's unknown what the controller
was doing while it was failing or how long it was failing for?
Recovery can be difficult.

Try using btrfs-find-root to find older roots, and use that value with
btrfs restore -t option. These are not as tidy as snapshots though,
the older they are, the more they dead end into more recent
overwrites. So you want to start out with the most recent roots you
can and work backwards in time.


-- 
Chris Murphy

Re: All files are damaged after btrfs restore

[Chris Murphy]

On Wed, Feb 24, 2021 at 10:40 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> I think you best chance is to start out trying to restore from a
> recent snapshot. As long as the failed controller wasn't writing
> totally spurious data in random locations, that snapshot should be
> intact.

i.e. the strategy for this is btrfs restore -r option

That only takes subvolid. You can get a subvolid listing with -l
option but this doesn't show the subvolume names yet (patch is
pending)
https://github.com/kdave/btrfs-progs/issues/289

As an alternative to applying that and building yourself, you can
approximate it with:

sudo btrfs insp dump-t -t 1 /dev/sda6 | grep -A 1 ROOT_REF

e.g.
    item 9 key (FS_TREE ROOT_REF 631) itemoff 14799 itemsize 26
        root ref key dirid 256 sequence 54 name varlog34

The subvolume varlog34 is subvolid 631. It's the same for snapshots.
So the restore command will use -r 631 to restore only from that
subvolume.


-- 
Chris Murphy

Re: All files are damaged after btrfs restore

[Sebastian Roller]

> > I think you best chance is to start out trying to restore from a
> > recent snapshot. As long as the failed controller wasn't writing
> > totally spurious data in random locations, that snapshot should be
> > intact.
>
> i.e. the strategy for this is btrfs restore -r option
>
> That only takes subvolid. You can get a subvolid listing with -l
> option but this doesn't show the subvolume names yet (patch is
> pending)
> https://github.com/kdave/btrfs-progs/issues/289
>
> As an alternative to applying that and building yourself, you can
> approximate it with:
>
> sudo btrfs insp dump-t -t 1 /dev/sda6 | grep -A 1 ROOT_REF
>
> e.g.
>     item 9 key (FS_TREE ROOT_REF 631) itemoff 14799 itemsize 26
>         root ref key dirid 256 sequence 54 name varlog34
>

Using this command I got a complete list of all the snapshots back to
2016 with full name.
I tried to restore from different snapshots and using btrfs restore -t
from some other older roots.
Unfortunately no matter which root I restore from, the files are
always the same. I selected a list of some larger files, namely ppts
and sgmls from one of our own tools, and restored them from different
roots. Then I compared the files by checksums. They are the same from
all roots I could find the files.
The output of btrfs restore gives me some errors for checksums and
deflate, but most of the files are just listed as restored.

Errors look like this:

Restoring /mnt/dumpo/recover/transfer/Hardware_Software/ABAQUS/AWI/AWI_6.14-2_2015.zip
Restoring /mnt/dumpo/recover/transfer/Hardware_Software/ABAQUS/AWI/installInstructions.txt
Done searching /Hardware_Software/ABAQUS/AWI
checksum verify failed on 57937054842880 found 000000B6 wanted 00000000
ERROR: lzo decompress failed: -4
Error copying data for
/mnt/dumpo/recover/transfer/Hardware_Software/ABAQUS/CM/CMA_win86_32_2012.0928.3/setup.exe
Error searching
/mnt/dumpo/recover/transfer/Hardware_Software/ABAQUS/CM/CMA_win86_32_2012.0928.3/setup.exe
ERROR: lzo decompress failed: -4
Error copying data for
/mnt/dumpo/recover/transfer/Hardware_Software/ABAQUS/CM/CMAInstaller.msi
ERROR: lzo decompress failed: -4
Error copying data for
/mnt/dumpo/recover/transfer/Hardware_Software/ABAQUS/CM/setup.exe
Error searching
/mnt/dumpo/recover/transfer/Hardware_Software/ABAQUS/CM/setup.exe

Most of the files are just listed as "Restoring ...". Still they are
severely damaged afterwards. They seem to contain "holes" filled with
0x00 (this is from some rudimentary hexdump examination of the files.)

Any chance to recover/restore from that? Thanks.

Sebastian

Re: All files are damaged after btrfs restore

[Chris Murphy]

On Fri, Feb 26, 2021 at 9:01 AM Sebastian Roller
<sebastian.roller@gmail.com> wrote:
>
> > > I think you best chance is to start out trying to restore from a
> > > recent snapshot. As long as the failed controller wasn't writing
> > > totally spurious data in random locations, that snapshot should be
> > > intact.
> >
> > i.e. the strategy for this is btrfs restore -r option
> >
> > That only takes subvolid. You can get a subvolid listing with -l
> > option but this doesn't show the subvolume names yet (patch is
> > pending)
> > https://github.com/kdave/btrfs-progs/issues/289
> >
> > As an alternative to applying that and building yourself, you can
> > approximate it with:
> >
> > sudo btrfs insp dump-t -t 1 /dev/sda6 | grep -A 1 ROOT_REF
> >
> > e.g.
> >     item 9 key (FS_TREE ROOT_REF 631) itemoff 14799 itemsize 26
> >         root ref key dirid 256 sequence 54 name varlog34
> >
>
> Using this command I got a complete list of all the snapshots back to
> 2016 with full name.
> I tried to restore from different snapshots and using btrfs restore -t
> from some other older roots.
> Unfortunately no matter which root I restore from, the files are
> always the same. I selected a list of some larger files, namely ppts
> and sgmls from one of our own tools, and restored them from different
> roots. Then I compared the files by checksums. They are the same from
> all roots I could find the files.
> The output of btrfs restore gives me some errors for checksums and
> deflate, but most of the files are just listed as restored.
>
> Errors look like this:
>
> Restoring /mnt/dumpo/recover/transfer/Hardware_Software/ABAQUS/AWI/AWI_6.14-2_2015.zip
> Restoring /mnt/dumpo/recover/transfer/Hardware_Software/ABAQUS/AWI/installInstructions.txt
> Done searching /Hardware_Software/ABAQUS/AWI
> checksum verify failed on 57937054842880 found 000000B6 wanted 00000000
> ERROR: lzo decompress failed: -4
> Error copying data for
> /mnt/dumpo/recover/transfer/Hardware_Software/ABAQUS/CM/CMA_win86_32_2012.0928.3/setup.exe
> Error searching
> /mnt/dumpo/recover/transfer/Hardware_Software/ABAQUS/CM/CMA_win86_32_2012.0928.3/setup.exe
> ERROR: lzo decompress failed: -4
> Error copying data for
> /mnt/dumpo/recover/transfer/Hardware_Software/ABAQUS/CM/CMAInstaller.msi
> ERROR: lzo decompress failed: -4
> Error copying data for
> /mnt/dumpo/recover/transfer/Hardware_Software/ABAQUS/CM/setup.exe
> Error searching
> /mnt/dumpo/recover/transfer/Hardware_Software/ABAQUS/CM/setup.exe
>
> Most of the files are just listed as "Restoring ...". Still they are
> severely damaged afterwards. They seem to contain "holes" filled with
> 0x00 (this is from some rudimentary hexdump examination of the files.)
>
> Any chance to recover/restore from that? Thanks.

I don't know. The exact nature of the damage of a failing controller
is adding a significant unknown component to it. If it was just a
matter of not writing anything at all, then there'd be no problem. But
it sounds like it wrote spurious or corrupt data, possibly into
locations that weren't even supposed to be written to.

I think if the snapshot b-tree is ok, and the chunk b-tree is ok, then
it should be possible to recover the data correctly without needing
any other tree. I'm not sure if that's how btrfs restore already
works.

Kernel 5.11 has a new feature, mount -o ro,rescue=all that is more
tolerant of mounting when there are various kinds of problems. But
there's another thread where a failed controller is thwarting
recovery, and that code is being looked at for further enhancement.
https://lore.kernel.org/linux-btrfs/CAEg-Je-DJW3saYKA2OBLwgyLU6j0JOF7NzXzECi0HJ5hft_5=A@mail.gmail.com/



--
Chris Murphy

Re: All files are damaged after btrfs restore

[Sebastian Roller]

> I don't know. The exact nature of the damage of a failing controller
> is adding a significant unknown component to it. If it was just a
> matter of not writing anything at all, then there'd be no problem. But
> it sounds like it wrote spurious or corrupt data, possibly into
> locations that weren't even supposed to be written to.

Unfortunately I cannot figure out exactly what happened. Logs end
Friday night while the backup script was running -- which also
includes a finalizing balancing of the device. Monday morning after
some exchange of hardware the machine came up being unable to mount
the device.

> I think if the snapshot b-tree is ok, and the chunk b-tree is ok, then
> it should be possible to recover the data correctly without needing
> any other tree. I'm not sure if that's how btrfs restore already
> works.
>
> Kernel 5.11 has a new feature, mount -o ro,rescue=all that is more
> tolerant of mounting when there are various kinds of problems. But
> there's another thread where a failed controller is thwarting
> recovery, and that code is being looked at for further enhancement.
> https://lore.kernel.org/linux-btrfs/CAEg-Je-DJW3saYKA2OBLwgyLU6j0JOF7NzXzECi0HJ5hft_5=A@mail.gmail.com/

OK -- I now had the chance to temporarily switch to 5.11.2. Output
looks cleaner, but the error stays the same.

root@hikitty:/mnt$ mount -o ro,rescue=all /dev/sdi1 hist/

[ 3937.815083] BTRFS info (device sdi1): enabling all of the rescue options
[ 3937.815090] BTRFS info (device sdi1): ignoring data csums
[ 3937.815093] BTRFS info (device sdi1): ignoring bad roots
[ 3937.815095] BTRFS info (device sdi1): disabling log replay at mount time
[ 3937.815098] BTRFS info (device sdi1): disk space caching is enabled
[ 3937.815100] BTRFS info (device sdi1): has skinny extents
[ 3938.903454] BTRFS error (device sdi1): bad tree block start, want
122583416078336 have 0
[ 3938.994662] BTRFS error (device sdi1): bad tree block start, want
99593231630336 have 0
[ 3939.201321] BTRFS error (device sdi1): bad tree block start, want
124762809384960 have 0
[ 3939.221395] BTRFS error (device sdi1): bad tree block start, want
124762809384960 have 0
[ 3939.221476] BTRFS error (device sdi1): failed to read block groups: -5
[ 3939.268928] BTRFS error (device sdi1): open_ctree failed

I still hope that there might be some error in the fs created by the
crash, which can be resolved instead of real damage to all the data in
the FS trees. I used a lot of snapshots and deduplication on that
device, so that I expect some damage by a hardware error. But I find
it hard to believe that every file got damaged.

Sebastian

Re: All files are damaged after btrfs restore

[Chris Murphy]

On Thu, Mar 4, 2021 at 8:35 AM Sebastian Roller
<sebastian.roller@gmail.com> wrote:
>
> > I don't know. The exact nature of the damage of a failing controller
> > is adding a significant unknown component to it. If it was just a
> > matter of not writing anything at all, then there'd be no problem. But
> > it sounds like it wrote spurious or corrupt data, possibly into
> > locations that weren't even supposed to be written to.
>
> Unfortunately I cannot figure out exactly what happened. Logs end
> Friday night while the backup script was running -- which also
> includes a finalizing balancing of the device. Monday morning after
> some exchange of hardware the machine came up being unable to mount
> the device.

It's probably not discernible with logs anyway. What hardware does
when it goes berserk? It's chaos. And all file systems have write
order requirements. It's fine if at a certain point writes just
abruptly stop going to stable media. But if things are written out of
order, or if the hardware acknowledges critical metadata writes are
written but were actually dropped, it's bad. For all file systems.


> OK -- I now had the chance to temporarily switch to 5.11.2. Output
> looks cleaner, but the error stays the same.
>
> root@hikitty:/mnt$ mount -o ro,rescue=all /dev/sdi1 hist/
>
> [ 3937.815083] BTRFS info (device sdi1): enabling all of the rescue options
> [ 3937.815090] BTRFS info (device sdi1): ignoring data csums
> [ 3937.815093] BTRFS info (device sdi1): ignoring bad roots
> [ 3937.815095] BTRFS info (device sdi1): disabling log replay at mount time
> [ 3937.815098] BTRFS info (device sdi1): disk space caching is enabled
> [ 3937.815100] BTRFS info (device sdi1): has skinny extents
> [ 3938.903454] BTRFS error (device sdi1): bad tree block start, want
> 122583416078336 have 0
> [ 3938.994662] BTRFS error (device sdi1): bad tree block start, want
> 99593231630336 have 0
> [ 3939.201321] BTRFS error (device sdi1): bad tree block start, want
> 124762809384960 have 0
> [ 3939.221395] BTRFS error (device sdi1): bad tree block start, want
> 124762809384960 have 0
> [ 3939.221476] BTRFS error (device sdi1): failed to read block groups: -5
> [ 3939.268928] BTRFS error (device sdi1): open_ctree failed

This looks like a super is expecting something that just isn't there
at all. If spurious behavior lasted only briefly during the hardware
failure, there's a chance of recovery. But this diminishes greatly if
the chaotic behavior was on-going for a while, many seconds or a few
minutes.


> I still hope that there might be some error in the fs created by the
> crash, which can be resolved instead of real damage to all the data in
> the FS trees. I used a lot of snapshots and deduplication on that
> device, so that I expect some damage by a hardware error. But I find
> it hard to believe that every file got damaged.

Correct. They aren't actually damaged.

However, there's maybe 5-15 MiB of critical metadata on Btrfs, and if
it gets corrupt, the keys to the maze are lost. And it becomes
difficult, sometimes impossible, to "bootstrap" the file system. There
are backup entry points, but depending on the workload, they go stale
in seconds to a few minutes, and can be subject to being overwritten.

When 'btrfs restore' is doing partial recovery that ends up with a lot
of damage and holes tells me it's found stale parts of the file system
- it's on old rails so to speak, there's nothing available to tell it
that this portion of the tree is just old and not valid anymore (or
only partially valid), but also the restore code is designed to be
more tolerant of errors because otherwise it would just do nothing at
all.

I think if you're able to find the most recent root node for a
snapshot you want to restore, along with an intact chunk tree it
should be possible to get data out of that snapshot. The difficulty is
finding it, because it could be almost anywhere.

OK so you said there's an original and backup file system, are they
both in equally bad shape, having been on the same controller? Are
they both btrfs?

What do you get for

btrfs insp dump-s -f /dev/sdXY

There might be a backup tree root in there that can be used with btrfs
restore -t

Also, sometimes easier to do this on  IRC on freenode.net in the channel #btrfs


-- 
Chris Murphy

Re: All files are damaged after btrfs restore

[Sebastian Roller]

> > > I don't know. The exact nature of the damage of a failing controller
> > > is adding a significant unknown component to it. If it was just a
> > > matter of not writing anything at all, then there'd be no problem. But
> > > it sounds like it wrote spurious or corrupt data, possibly into
> > > locations that weren't even supposed to be written to.
> >
> > Unfortunately I cannot figure out exactly what happened. Logs end
> > Friday night while the backup script was running -- which also
> > includes a finalizing balancing of the device. Monday morning after
> > some exchange of hardware the machine came up being unable to mount
> > the device.
>
> It's probably not discernible with logs anyway. What hardware does
> when it goes berserk? It's chaos. And all file systems have write
> order requirements. It's fine if at a certain point writes just
> abruptly stop going to stable media. But if things are written out of
> order, or if the hardware acknowledges critical metadata writes are
> written but were actually dropped, it's bad. For all file systems.
>
>
> > OK -- I now had the chance to temporarily switch to 5.11.2. Output
> > looks cleaner, but the error stays the same.
> >
> > root@hikitty:/mnt$ mount -o ro,rescue=all /dev/sdi1 hist/
> >
> > [ 3937.815083] BTRFS info (device sdi1): enabling all of the rescue options
> > [ 3937.815090] BTRFS info (device sdi1): ignoring data csums
> > [ 3937.815093] BTRFS info (device sdi1): ignoring bad roots
> > [ 3937.815095] BTRFS info (device sdi1): disabling log replay at mount time
> > [ 3937.815098] BTRFS info (device sdi1): disk space caching is enabled
> > [ 3937.815100] BTRFS info (device sdi1): has skinny extents
> > [ 3938.903454] BTRFS error (device sdi1): bad tree block start, want
> > 122583416078336 have 0
> > [ 3938.994662] BTRFS error (device sdi1): bad tree block start, want
> > 99593231630336 have 0
> > [ 3939.201321] BTRFS error (device sdi1): bad tree block start, want
> > 124762809384960 have 0
> > [ 3939.221395] BTRFS error (device sdi1): bad tree block start, want
> > 124762809384960 have 0
> > [ 3939.221476] BTRFS error (device sdi1): failed to read block groups: -5
> > [ 3939.268928] BTRFS error (device sdi1): open_ctree failed
>
> This looks like a super is expecting something that just isn't there
> at all. If spurious behavior lasted only briefly during the hardware
> failure, there's a chance of recovery. But this diminishes greatly if
> the chaotic behavior was on-going for a while, many seconds or a few
> minutes.
>
>
> > I still hope that there might be some error in the fs created by the
> > crash, which can be resolved instead of real damage to all the data in
> > the FS trees. I used a lot of snapshots and deduplication on that
> > device, so that I expect some damage by a hardware error. But I find
> > it hard to believe that every file got damaged.
>
> Correct. They aren't actually damaged.
>
> However, there's maybe 5-15 MiB of critical metadata on Btrfs, and if
> it gets corrupt, the keys to the maze are lost. And it becomes
> difficult, sometimes impossible, to "bootstrap" the file system. There
> are backup entry points, but depending on the workload, they go stale
> in seconds to a few minutes, and can be subject to being overwritten.
>
> When 'btrfs restore' is doing partial recovery that ends up with a lot
> of damage and holes tells me it's found stale parts of the file system
> - it's on old rails so to speak, there's nothing available to tell it
> that this portion of the tree is just old and not valid anymore (or
> only partially valid), but also the restore code is designed to be
> more tolerant of errors because otherwise it would just do nothing at
> all.
>
> I think if you're able to find the most recent root node for a
> snapshot you want to restore, along with an intact chunk tree it
> should be possible to get data out of that snapshot. The difficulty is
> finding it, because it could be almost anywhere.

Would it make sense to just try  restore -t on any root I got with
btrfs-find-root with all of the snapshots?

> OK so you said there's an original and backup file system, are they
> both in equally bad shape, having been on the same controller? Are
> they both btrfs?

The original / live file system was not btrfs but xfs. It is in a
different but equally bad state than the backup. We used bcache with a
write-back cache on a ssd which is now completely dead (does not get
recognized by any server anymore). To get the file system mounted I
ran xfs-repair. After that only 6% of the data was left and this is
nearly completely in lost+found. I'm now trying to sort these files by
type, since the data itself looks OK. Unfortunately the surviving
files seem to be the oldest ones.

> What do you get for
>
> btrfs insp dump-s -f /dev/sdXY
>
> There might be a backup tree root in there that can be used with btrfs
> restore -t

This is the output of ./btrfs  insp dump-s -f /dev/sdi1 run with
btrfs-progs 5.9.

./btrfs insp dump-s -f /dev/sdi1
superblock: bytenr=65536, device=/dev/sdi1
---------------------------------------------------------
csum_type               0 (crc32c)
csum_size               4
csum                    0x9e6891fc [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    56051c5f-fca6-4d54-a04e-1c1d8129fe56
metadata_uuid           56051c5f-fca6-4d54-a04e-1c1d8129fe56
label                   history
generation              825256
root                    122583415865344
sys_array_size          129
chunk_root_generation   825256
root_level              2
chunk_root              141944043454464
chunk_root_level        2
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             80013782134784
bytes_used              75176955760640
sectorsize              4096
nodesize                16384
leafsize (deprecated)   16384
stripesize              4096
root_dir                6
num_devices             1
compat_flags            0x0
compat_ro_flags         0x0
incompat_flags          0x169
                        ( MIXED_BACKREF |
                          COMPRESS_LZO |
                          BIG_METADATA |
                          EXTENDED_IREF |
                          SKINNY_METADATA )
cache_generation        825256
uuid_tree_generation    825256
dev_item.uuid           844e80b3-a8d5-4738-ac8a-4f54980556f6
dev_item.fsid           56051c5f-fca6-4d54-a04e-1c1d8129fe56 [match]
dev_item.type           0
dev_item.total_bytes    80013782134784
dev_item.bytes_used     75413317484544
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          2
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0
sys_chunk_array[2048]:
        item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 141944034426880)
                length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP
                io_align 65536 io_width 65536 sector_size 4096
                num_stripes 2 sub_stripes 1
                        stripe 0 devid 2 offset 2034741805056
                        dev_uuid 844e80b3-a8d5-4738-ac8a-4f54980556f6
                        stripe 1 devid 2 offset 2034775359488
                        dev_uuid 844e80b3-a8d5-4738-ac8a-4f54980556f6
backup_roots[4]:
        backup 0:
                backup_tree_root:       122583415865344 gen: 825256     level: 2
                backup_chunk_root:      141944043454464 gen: 825256     level: 2
                backup_extent_root:     122583418175488 gen: 825256     level: 3
                backup_fs_root:         58363985428480  gen: 789775     level: 0
                backup_dev_root:        122583415783424 gen: 825256     level: 1
                backup_csum_root:       122583553703936 gen: 825256     level: 3
                backup_total_bytes:     80013782134784
                backup_bytes_used:      75176955760640
                backup_num_devices:     1

        backup 1:
                backup_tree_root:       122343302234112 gen: 825253     level: 2
                backup_chunk_root:      141944034426880 gen: 825251     level: 2
                backup_extent_root:     122343333937152 gen: 825253     level: 3
                backup_fs_root:         58363985428480  gen: 789775     level: 0
                backup_dev_root:        122077274357760 gen: 825250     level: 1
                backup_csum_root:       122343380992000 gen: 825253     level: 3
                backup_total_bytes:     80013782134784
                backup_bytes_used:      75176955105280
                backup_num_devices:     1

        backup 2:
                backup_tree_root:       122343762804736 gen: 825254     level: 2
                backup_chunk_root:      141944034426880 gen: 825251     level: 2
                backup_extent_root:     122343762935808 gen: 825254     level: 3
                backup_fs_root:         58363985428480  gen: 789775     level: 0
                backup_dev_root:        122077274357760 gen: 825250     level: 1
                backup_csum_root:       122343764967424 gen: 825254     level: 3
                backup_total_bytes:     80013782134784
                backup_bytes_used:      75176955105280
                backup_num_devices:     1

        backup 3:
                backup_tree_root:       122574011269120 gen: 825255     level: 2
                backup_chunk_root:      141944034426880 gen: 825251     level: 2
                backup_extent_root:     122574011432960 gen: 825255     level: 3
                backup_fs_root:         58363985428480  gen: 789775     level: 0
                backup_dev_root:        122077274357760 gen: 825250     level: 1
                backup_csum_root:       122574014791680 gen: 825255     level: 3
                backup_total_bytes:     80013782134784
                backup_bytes_used:      75176955236352
                backup_num_devices:     1

Re: All files are damaged after btrfs restore

[Chris Murphy]

On Sun, Mar 7, 2021 at 6:58 AM Sebastian Roller
<sebastian.roller@gmail.com> wrote:
>
> Would it make sense to just try  restore -t on any root I got with
> btrfs-find-root with all of the snapshots?

Yes but I think you've tried this and you only got corrupt files or
files with holes, so that suggests very recent roots are just bad due
to the corruption, and older ones are pointing to a mix of valid and
stale blocks and it just ends up in confusion.

I think what you're after is 'btrfs restore -f'

       -f <bytenr>
           only restore files that are under specified subvolume root
pointed by <bytenr>

You can get this value from each 'tree root' a.k.a. the root of roots
tree, what the super calls simply 'root'. That contains references for
all the other trees' roots. For example:

    item 12 key (257 ROOT_ITEM 0) itemoff 12936 itemsize 439
        generation 97406 root_dirid 256 bytenr 30752768 level 1 refs 1
        lastsnap 93151 byte_limit 0 bytes_used 2818048 flags 0x0(none)
        uuid 4a0fa0d3-783c-bc42-bee1-ffcbe7325753
        ctransid 97406 otransid 7 stransid 0 rtransid 0
        ctime 1615103595.233916841 (2021-03-07 00:53:15)
        otime 1603562604.21506964 (2020-10-24 12:03:24)
        drop key (0 UNKNOWN.0 0) level 0
    item 13 key (257 ROOT_BACKREF 5) itemoff 12911 itemsize 25
        root backref key dirid 256 sequence 2 name newpool



The name of this subvolume is newpool, the subvolid is 257, and its
address is bytenr 30752768. That's the value to plug into btrfs
restore -f

The thing is, it needs an intact chunk tree, i.e. not damaged and not
too old, in order to translate that logical address into a physical
device and physical address.





>
> > OK so you said there's an original and backup file system, are they
> > both in equally bad shape, having been on the same controller? Are
> > they both btrfs?
>
> The original / live file system was not btrfs but xfs. It is in a
> different but equally bad state than the backup. We used bcache with a
> write-back cache on a ssd which is now completely dead (does not get
> recognized by any server anymore). To get the file system mounted I
> ran xfs-repair. After that only 6% of the data was left and this is
> nearly completely in lost+found. I'm now trying to sort these files by
> type, since the data itself looks OK. Unfortunately the surviving
> files seem to be the oldest ones.

Yeah writeback means the bcache device must survive and be healthy
before any repair attempts should be made, even restore attempts. It
also means you need hardware isolation, one SSD per HDD. Otherwise one
SSD failing means the whole thing falls apart. The mode to use for
read caching is writethrough.


>         backup 0:
>                 backup_tree_root:       122583415865344 gen: 825256     level: 2
>                 backup_chunk_root:      141944043454464 gen: 825256     level: 2


>         backup 1:
>                 backup_tree_root:       122343302234112 gen: 825253     level: 2
>                 backup_chunk_root:      141944034426880 gen: 825251     level: 2

>         backup 2:
>                 backup_tree_root:       122343762804736 gen: 825254     level: 2
>                 backup_chunk_root:      141944034426880 gen: 825251     level: 2

>         backup 3:
>                 backup_tree_root:       122574011269120 gen: 825255     level: 2
>                 backup_chunk_root:      141944034426880 gen: 825251     level: 2

OK this is interesting. There's two chunk trees to choose from. So is
the restore problem because older roots point to the older chunk tree
which is already going stale, and just isn't assembling blocks
correctly anymore? Or is it because the new chunk tree is bad?

On 72 TB, the last thing I want to recommend is chunk-recover. That'll
take forever but it'd be interesting to know which of these chunk
trees is good. The chunk tree is in the system block group. It's
pretty tiny so it's a small target for being overwritten...and it's
cow. So there isn't a reason to immediately start overwriting it. I'm
thinking maybe the new one got interrupted by the failure and the old
one is intact.

Ok so the next step is to find a snapshot you want to restore.

btrfs insp dump-t -t 1 /dev/sdi1

And you'll need to look for a snapshot name in there, find its bytenr,
and let's first see if just using that works. If it doesn't then maybe
combining it with the next most recent root tree will work.


-- 
Chris Murphy

Re: All files are damaged after btrfs restore

[Sebastian Roller]

> > Would it make sense to just try  restore -t on any root I got with
> > btrfs-find-root with all of the snapshots?
>
> Yes but I think you've tried this and you only got corrupt files or
> files with holes, so that suggests very recent roots are just bad due
> to the corruption, and older ones are pointing to a mix of valid and
> stale blocks and it just ends up in confusion.
>
> I think what you're after is 'btrfs restore -f'
>
>        -f <bytenr>
>            only restore files that are under specified subvolume root
> pointed by <bytenr>
>
> You can get this value from each 'tree root' a.k.a. the root of roots
> tree, what the super calls simply 'root'. That contains references for
> all the other trees' roots. For example:
>
>     item 12 key (257 ROOT_ITEM 0) itemoff 12936 itemsize 439
>         generation 97406 root_dirid 256 bytenr 30752768 level 1 refs 1
>         lastsnap 93151 byte_limit 0 bytes_used 2818048 flags 0x0(none)
>         uuid 4a0fa0d3-783c-bc42-bee1-ffcbe7325753
>         ctransid 97406 otransid 7 stransid 0 rtransid 0
>         ctime 1615103595.233916841 (2021-03-07 00:53:15)
>         otime 1603562604.21506964 (2020-10-24 12:03:24)
>         drop key (0 UNKNOWN.0 0) level 0
>     item 13 key (257 ROOT_BACKREF 5) itemoff 12911 itemsize 25
>         root backref key dirid 256 sequence 2 name newpool
>
>
>
> The name of this subvolume is newpool, the subvolid is 257, and its
> address is bytenr 30752768. That's the value to plug into btrfs
> restore -f

I found 12 of these 'tree roots' on the volume. All the snapshots are
under the same tree root. This seems to be the subvolume where I put
the snapshots. So for the snapshots there is only one option to use
with btrfs restore -r. But I also found the data I'm looking for under
some other of these tree roots. One of them is clearly the subvolume
the backup went to (the source of the snapshots). But there is also a
very old snapshot (4 years old) that has a tree root on its own. The
files I restored  from there are different -- regarding checksums.
They are also corrupted, but different. I have to do some more
hexdumps to figure out, if it's better.

> The thing is, it needs an intact chunk tree, i.e. not damaged and not
> too old, in order to translate that logical address into a physical
> device and physical address.

> > > OK so you said there's an original and backup file system, are they
> > > both in equally bad shape, having been on the same controller? Are
> > > they both btrfs?
> >
> > The original / live file system was not btrfs but xfs. It is in a
> > different but equally bad state than the backup. We used bcache with a
> > write-back cache on a ssd which is now completely dead (does not get
> > recognized by any server anymore). To get the file system mounted I
> > ran xfs-repair. After that only 6% of the data was left and this is
> > nearly completely in lost+found. I'm now trying to sort these files by
> > type, since the data itself looks OK. Unfortunately the surviving
> > files seem to be the oldest ones.
>
> Yeah writeback means the bcache device must survive and be healthy
> before any repair attempts should be made, even restore attempts. It
> also means you need hardware isolation, one SSD per HDD. Otherwise one
> SSD failing means the whole thing falls apart. The mode to use for
> read caching is writethrough.

Hmm. Lessons learned there. This server handles the home-directories
of our personal workstations on which we work on  quite large files (>
5 GiB). So combining write-caching with 2x 10GBit Ethernet was quite
good -- regarding performance.

>
> >         backup 0:
> >                 backup_tree_root:       122583415865344 gen: 825256     level: 2
> >                 backup_chunk_root:      141944043454464 gen: 825256     level: 2
>
>
> >         backup 1:
> >                 backup_tree_root:       122343302234112 gen: 825253     level: 2
> >                 backup_chunk_root:      141944034426880 gen: 825251     level: 2
>
> >         backup 2:
> >                 backup_tree_root:       122343762804736 gen: 825254     level: 2
> >                 backup_chunk_root:      141944034426880 gen: 825251     level: 2
>
> >         backup 3:
> >                 backup_tree_root:       122574011269120 gen: 825255     level: 2
> >                 backup_chunk_root:      141944034426880 gen: 825251     level: 2
>
> OK this is interesting. There's two chunk trees to choose from. So is
> the restore problem because older roots point to the older chunk tree
> which is already going stale, and just isn't assembling blocks
> correctly anymore? Or is it because the new chunk tree is bad?

Is there a way to choose the chunk tree I'm using for operations like
btrfs restore?

> On 72 TB, the last thing I want to recommend is chunk-recover. That'll
> take forever but it'd be interesting to know which of these chunk
> trees is good. The chunk tree is in the system block group. It's
> pretty tiny so it's a small target for being overwritten...and it's
> cow. So there isn't a reason to immediately start overwriting it. I'm
> thinking maybe the new one got interrupted by the failure and the old
> one is intact.

I already ran chunk-recover. It needs two days to finish. But I used
btrfs-tools version 4.14 and it failed.

root@hikitty:/mnt$ btrfs rescue chunk-recover /dev/sdf1
Scanning: DONE in dev0
checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
bytenr mismatch, want=124762809384960, have=0
open with broken chunk error
Chunk tree recovery failed

I could try again with a newer version. (?) Because with version 4.14
also btrfs restore failed.

> btrfs insp dump-t -t 1 /dev/sdi1
>
> And you'll need to look for a snapshot name in there, find its bytenr,
> and let's first see if just using that works. If it doesn't then maybe
> combining it with the next most recent root tree will work.

I am working backwards right now using btrfs restore -f in combination
with -t. So far no success.

Sebastian

Re: All files are damaged after btrfs restore

[Chris Murphy]

On Tue, Mar 9, 2021 at 10:03 AM Sebastian Roller
<sebastian.roller@gmail.com> wrote:

> I found 12 of these 'tree roots' on the volume. All the snapshots are
> under the same tree root. This seems to be the subvolume where I put
> the snapshots.

Snapshots are subvolumes. All of them will appear in the root tree,
even if they're organized as being in a directory or in some other
subvolume.

>So for the snapshots there is only one option to use
> with btrfs restore -r.

It can be done by its own root node address using -f or by subvolid
using -r. The latter needs to be looked up in a reliable root tree.
But I think the distinction may not matter here because really it's
the chunk tree that's messed up, and that's what's used to find
everything. The addresses in the file tree (the subvolume/snapshot
tree that contains file listings, inodes, metadata, and the address of
the file) are all logical addresses in btrfs linear space. That means
nothing without the translation to physical device and blocks, which
is the job of the chunk tree.

>But I also found the data I'm looking for under
> some other of these tree roots. One of them is clearly the subvolume
> the backup went to (the source of the snapshots). But there is also a
> very old snapshot (4 years old) that has a tree root on its own. The
> files I restored  from there are different -- regarding checksums.
> They are also corrupted, but different. I have to do some more
> hexdumps to figure out, if it's better.

Unfortunately when things are messed up badly, the recovery tools may
be looking at a wrong or partial checksum tree and it just spits out
checksum complaints as a matter of course. You'd have to inspect the
file contents themselves, the checksum warnings might be real or
bogus.
> > OK this is interesting. There's two chunk trees to choose from. So is
> > the restore problem because older roots point to the older chunk tree
> > which is already going stale, and just isn't assembling blocks
> > correctly anymore? Or is it because the new chunk tree is bad?
>
> Is there a way to choose the chunk tree I'm using for operations like
> btrfs restore?

Looks like the answer is no. The chunk tree really has to be correct
first before anything else because it's central to doing all the
logical to physical address translation. And if it's busted and can't
be repaired then nothing else is likely to work or be repairable. It's
that critical.



> I already ran chunk-recover. It needs two days to finish. But I used
> btrfs-tools version 4.14 and it failed.

I'd have to go dig in git history to even know if there's been
improvements in chunk recover since then. But I pretty much consider
any file system's tool obsolete within a year. I think it's total
nonsense that distributions are intentionally using old tools.

>
> root@hikitty:/mnt$ btrfs rescue chunk-recover /dev/sdf1
> Scanning: DONE in dev0
> checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
> checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
> checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> bytenr mismatch, want=124762809384960, have=0
> open with broken chunk error
> Chunk tree recovery failed
>
> I could try again with a newer version. (?) Because with version 4.14
> also btrfs restore failed.

It is entirely possible that 5.11 fails exactly the same way because
it's just too badly damaged for the current state of the recovery
tools to deal with damage of this kind. But it's also possible it'll
work. It's a coin toss unless someone else a lot more familiar with
the restore code speaks up. But looking at just the summary change
log, it looks like no work has happened in chunk recover for a while.

https://btrfs.wiki.kernel.org/index.php/Changelog


> > btrfs insp dump-t -t 1 /dev/sdi1
> >
> > And you'll need to look for a snapshot name in there, find its bytenr,
> > and let's first see if just using that works. If it doesn't then maybe
> > combining it with the next most recent root tree will work.
>
> I am working backwards right now using btrfs restore -f in combination
> with -t. So far no success.

Yep. I think it comes down to the chunk tree needing to be reasonable
first, before anything else is possible.


-- 
Chris Murphy

Re: All files are damaged after btrfs restore

[Sebastian Roller]

Hi again.

> Looks like the answer is no. The chunk tree really has to be correct
> first before anything else because it's central to doing all the
> logical to physical address translation. And if it's busted and can't
> be repaired then nothing else is likely to work or be repairable. It's
> that critical.
>
> > I already ran chunk-recover. It needs two days to finish. But I used
> > btrfs-tools version 4.14 and it failed.
>
> I'd have to go dig in git history to even know if there's been
> improvements in chunk recover since then. But I pretty much consider
> any file system's tool obsolete within a year. I think it's total
> nonsense that distributions are intentionally using old tools.
> >
> > root@hikitty:/mnt$ btrfs rescue chunk-recover /dev/sdf1
> > Scanning: DONE in dev0
> > checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
> > checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
> > checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> > checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> > checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> > checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> > bytenr mismatch, want=124762809384960, have=0
> > open with broken chunk error
> > Chunk tree recovery failed
> >
> > I could try again with a newer version. (?) Because with version 4.14
> > also btrfs restore failed.
>
> It is entirely possible that 5.11 fails exactly the same way because
> it's just too badly damaged for the current state of the recovery
> tools to deal with damage of this kind. But it's also possible it'll
> work. It's a coin toss unless someone else a lot more familiar with
> the restore code speaks up. But looking at just the summary change
> log, it looks like no work has happened in chunk recover for a while.
>
> https://btrfs.wiki.kernel.org/index.php/Changelog

So I ran another chunk-recover with btrfs-progs version 5.11. This is
part of the output. (The list doesn't allow me attach the whole output
to this mail (5 mb zipped). But if you let me know what's important I
can send that.)

root@hikitty:~$ nohup /root/install/btrfs-progs-5.11/btrfs -v rescue
chunk-recover /dev/sdi1 >
/transfer/sebroll/btrfs-rescue-chunk-recover.out.txt 2>&1 &
nohup: ignoring input
All Devices:
        Device: id = 2, name = /dev/sdi1
Scanning: DONE in dev0

DEVICE SCAN RESULT:
Filesystem Information:
        sectorsize: 4096
        nodesize: 16384
        tree root generation: 825256
        chunk root generation: 825256

All Devices:
        Device: id = 2, name = /dev/sdi1

All Block Groups:
        Block Group: start = 49477515739136, len = 1073741824, flag = 1
        Block Group: start = 49478589480960, len = 1073741824, flag = 1
(…)
       Block Group: start = 141942960685056, len = 1073741824, flag = 1
       Block Group: start = 141944034426880, len = 33554432, flag = 22

All Chunks:
        Chunk: start = 49477515739136, len = 1073741824, type = 1,
num_stripes = 1
            Stripes list:
            [ 0] Stripe: devid = 2, offset = 1048576
        Chunk: start = 49478589480960, len = 1073741824, type = 1,
num_stripes = 1
            Stripes list:
            [ 0] Stripe: devid = 2, offset = 1074790400
(…)
        Chunk: start = 141942960685056, len = 1073741824, type = 1,
num_stripes = 1
            Stripes list:
            [ 0] Stripe: devid = 2, offset = 75414325166080
        Chunk: start = 141944034426880, len = 33554432, type = 22,
num_stripes = 2
            Stripes list:
            [ 0] Stripe: devid = 2, offset = 2034741805056
            [ 1] Stripe: devid = 2, offset = 2034775359488

All Device Extents:
        Device extent: devid = 2, start = 1048576, len = 1073741824,
chunk offset = 49477515739136
        Device extent: devid = 2, start = 1074790400, len =
1073741824, chunk offset = 49478589480960
        Device extent: devid = 2, start = 2148532224, len =
1073741824, chunk offset = 49479663222784
(…)
        Device extent: devid = 2, start = 75412177682432, len =
1073741824, chunk offset = 141940813201408
        Device extent: devid = 2, start = 75413251424256, len =
1073741824, chunk offset = 141941886943232
        Device extent: devid = 2, start = 75414325166080, len =
1073741824, chunk offset = 141942960685056

CHECK RESULT:
Recoverable Chunks:
  Chunk: start = 49477515739136, len = 1073741824, type = 1, num_stripes = 1
      Stripes list:
      [ 0] Stripe: devid = 2, offset = 1048576
      Block Group: start = 49477515739136, len = 1073741824, flag = 1
      Device extent list:
          [ 0]Device extent: devid = 2, start = 1048576, len =
1073741824, chunk offset = 49477515739136
  Chunk: start = 49478589480960, len = 1073741824, type = 1, num_stripes = 1
      Stripes list:
      [ 0] Stripe: devid = 2, offset = 1074790400
      Block Group: start = 49478589480960, len = 1073741824, flag = 1
      Device extent list:
          [ 0]Device extent: devid = 2, start = 1074790400, len =
1073741824, chunk offset = 49478589480960
  Chunk: start = 49479663222784, len = 1073741824, type = 1, num_stripes = 1
      Stripes list:
      [ 0] Stripe: devid = 2, offset = 2148532224
      Block Group: start = 49479663222784, len = 1073741824, flag = 1
      Device extent list:
          [ 0]Device extent: devid = 2, start = 2148532224, len =
1073741824, chunk offset = 49479663222784
(…)
  Chunk: start = 141085812719616, len = 1073741824, type = 1, num_stripes = 1
      Stripes list:
      [ 0] Stripe: devid = 2, offset = 74690623176704
      No block group.
      Device extent list:
          [ 0]Device extent: devid = 2, start = 74690623176704, len =
1073741824, chunk offset = 141085812719616
  Chunk: start = 141403740962816, len = 1073741824, type = 1, num_stripes = 1
      Stripes list:
      [ 0] Stripe: devid = 2, offset = 73223891845120
      No block group.
      Device extent list:
          [ 0]Device extent: devid = 2, start = 73223891845120, len =
1073741824, chunk offset = 141403740962816
Unrecoverable Chunks:
  Chunk: start = 73953460617216, len = 1073741824, type = 1, num_stripes = 1
      Stripes list:
      [ 0] Stripe: devid = 2, offset = 23810225995776
      No block group.
      No device extent.
  Chunk: start = 75698291081216, len = 1073741824, type = 1, num_stripes = 1
      Stripes list:
      [ 0] Stripe: devid = 2, offset = 25555056459776
      No block group.
      No device extent.
(…)
  Chunk: start = 139435974852608, len = 1073741824, type = 1, num_stripes = 1
      Stripes list:
      [ 0] Stripe: devid = 2, offset = 49594055524352
      Block Group: start = 139435974852608, len = 1073741824, flag = 1
      No device extent.
  Chunk: start = 140101996773376checksum verify failed on
99593231630336 found 000000B6 wanted 00000000
checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
bad tree block 124762809384960, bytenr mismatch, want=124762809384960, have=0
ERROR: failed to read block groups: Input/output error
open with broken chunk error
, len = 1073741824, type = 1, num_stripes = 1
      Stripes list:
      [ 0] Stripe: devid = 2, offset = 58501817696256
      Block Group: start = 140101996773376, len = 1073741824, flag = 1
      No device extent.
  Chunk: start = 140221215670272, len = 1073741824, type = 1, num_stripes = 1
      Stripes list:
      [ 0] Stripe: devid = 2, offset = 58416992092160
      Block Group: start = 140221215670272, len = 1073741824, flag = 1
      No device extent.
(…)
  Chunk: start = 141836593135616, len = 33554432, type = 22, num_stripes = 2
      Stripes list:
      [ 0] Stripe: devid = 2, offset = 2034741805056
      [ 1] Stripe: devid = 2, offset = 2034775359488
      No block group.
      No device extent.
  Chunk: start = 141269456125952, len = 33554432, type = 22, num_stripes = 0
      Stripes list:
      Block Group: start = 141269456125952, len = 33554432, flag = 22
      No device extent.

Total Chunks:           72859
  Recoverable:          68784
  Unrecoverable:        4075

Orphan Block Groups:

Orphan Device Extents:

Chunk tree recovery failed



So. Is this good or bad? First, there are a lot of unrecoverable
chunks. And some of them are system chunks.
But my biggest issue seems to be that "bad tree block" error that is
showing up and which I'm getting any time I try to access the
file-system. This error also seems to prevent mounting and btrfs
check. Can I use the position given there to look up the corresponding
chunk? If so:

Block Group and Chunks around bad tree block 124762809384960:

        Block Group: start = 124760809799680, len = 1073741824, flag = 1
        Block Group: start = 124761883541504, len = 1073741824, flag = 24
        Block Group: start = 124798390763520, len = 1073741824, flag =
1

        Chunk: start = 124760809799680, len = 1073741824, type = 1,
num_stripes = 1
            Stripes list:
            [ 0] Stripe: devid = 2, offset = 67164766732288
        Chunk: start = 124761883541504, len = 1073741824, type = 24,
num_stripes = 2
            Stripes list:
            [ 0] Stripe: devid = 2, offset = 67165840474112
            [ 1] Stripe: devid = 2, offset = 67166914215936
        Chunk: start = 124777989668864, len = 1073741824, type = 1,
num_stripes = 1
            Stripes list:
            [ 0] Stripe: devid = 2, offset = 67183020343296

The bad tree block seems to contain meta data. Can I somehow recover
this? It seems to be DUP (2 Stripes)…

> > > And you'll need to look for a snapshot name in there, find its bytenr,
> > > and let's first see if just using that works. If it doesn't then maybe
> > > combining it with the next most recent root tree will work.
> >
> > I am working backwards right now using btrfs restore -f in combination
> > with -t. So far no success.
>
> Yep. I think it comes down to the chunk tree needing to be reasonable
> first, before anything else is possible.

Now I tried every possible combination with btrfs restore.
Unfortunately I found no working solution to restore the data
undamaged. As mentioned before the files contain "holes" of 0x00.
These holes are exactly 4 MiB in size. Does this tell anything?
Any ideas how to proceed from here on? Accept the data loss and try to
repair the files as good as possible -- putting them together from
different saves (if available) is a real grind. Or professional data
recovery -- but wouldn't they face the same issue regarding the
chunk-tree?

Kind regards,
Sebastian

Re: All files are damaged after btrfs restore

[Chris Murphy]

Hi,

The problem exceeds my knowledge of both Btrfs and bcache/ssd failure
modes. I'm not sure what professional data recovery can really do,
other than throw a bunch of people at stitching things back together
again without any help from the file system. I know that the state of
the repair tools is not great, and it is confusing what to use in what
order. I don't know if a support contract from one of the distros
supporting Btrfs (most likely SUSE) is a better way to get assistance
with this kind of recovery while also supporting development. But
that's a question for SUSE sales :)

Most of the emphasis of upstream development has been on preventing
problems from happening to critical Btrfs metadata in the first place.
Its ability to self-heal really depends on it having independent block
devices to write to, e.g. metadata raid 1. Metadata DUP might normally
help with only spinning drives, but with a cache device, it's going to
cache all of these concurrent metadata writes.

If critical metadata is seriously damaged or missing, it's probably
impossible to fix or even skip over with the current state of the
tools. Current code needs an entry point into the chunk tree in order
to make the logical to physical mapping; and then needs an entry point
to the root tree to get to the proper snapshot file tree. If all the
recent and critical metadata is lost on the failed bcache caching
device, then a totally different strategy is needed.

The file btree for the snapshot you want should be on the backing
device, as well as its data chunks, and the mapping in the ~94% of the
chunk tree that's on disk. I won't be surprised if the file system is
broken beyond repair, but I'd be a little surprised if someone more
knowledgeable can't figure out a way to get the data out of a week old
snapshot. But that's speculation on my part. I really have no idea how
long it could take for bcache in writeback mode to flush to the
backing device.

--
Chris Murphy


On Tue, Mar 16, 2021 at 3:35 AM Sebastian Roller
<sebastian.roller@gmail.com> wrote:
>
> Hi again.
>
> > Looks like the answer is no. The chunk tree really has to be correct
> > first before anything else because it's central to doing all the
> > logical to physical address translation. And if it's busted and can't
> > be repaired then nothing else is likely to work or be repairable. It's
> > that critical.
> >
> > > I already ran chunk-recover. It needs two days to finish. But I used
> > > btrfs-tools version 4.14 and it failed.
> >
> > I'd have to go dig in git history to even know if there's been
> > improvements in chunk recover since then. But I pretty much consider
> > any file system's tool obsolete within a year. I think it's total
> > nonsense that distributions are intentionally using old tools.
> > >
> > > root@hikitty:/mnt$ btrfs rescue chunk-recover /dev/sdf1
> > > Scanning: DONE in dev0
> > > checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
> > > checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
> > > checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> > > checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> > > checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> > > checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> > > bytenr mismatch, want=124762809384960, have=0
> > > open with broken chunk error
> > > Chunk tree recovery failed
> > >
> > > I could try again with a newer version. (?) Because with version 4.14
> > > also btrfs restore failed.
> >
> > It is entirely possible that 5.11 fails exactly the same way because
> > it's just too badly damaged for the current state of the recovery
> > tools to deal with damage of this kind. But it's also possible it'll
> > work. It's a coin toss unless someone else a lot more familiar with
> > the restore code speaks up. But looking at just the summary change
> > log, it looks like no work has happened in chunk recover for a while.
> >
> > https://btrfs.wiki.kernel.org/index.php/Changelog
>
> So I ran another chunk-recover with btrfs-progs version 5.11. This is
> part of the output. (The list doesn't allow me attach the whole output
> to this mail (5 mb zipped). But if you let me know what's important I
> can send that.)
>
> root@hikitty:~$ nohup /root/install/btrfs-progs-5.11/btrfs -v rescue
> chunk-recover /dev/sdi1 >
> /transfer/sebroll/btrfs-rescue-chunk-recover.out.txt 2>&1 &
> nohup: ignoring input
> All Devices:
>         Device: id = 2, name = /dev/sdi1
> Scanning: DONE in dev0
>
> DEVICE SCAN RESULT:
> Filesystem Information:
>         sectorsize: 4096
>         nodesize: 16384
>         tree root generation: 825256
>         chunk root generation: 825256
>
> All Devices:
>         Device: id = 2, name = /dev/sdi1
>
> All Block Groups:
>         Block Group: start = 49477515739136, len = 1073741824, flag = 1
>         Block Group: start = 49478589480960, len = 1073741824, flag = 1
> (…)
>        Block Group: start = 141942960685056, len = 1073741824, flag = 1
>        Block Group: start = 141944034426880, len = 33554432, flag = 22
>
> All Chunks:
>         Chunk: start = 49477515739136, len = 1073741824, type = 1,
> num_stripes = 1
>             Stripes list:
>             [ 0] Stripe: devid = 2, offset = 1048576
>         Chunk: start = 49478589480960, len = 1073741824, type = 1,
> num_stripes = 1
>             Stripes list:
>             [ 0] Stripe: devid = 2, offset = 1074790400
> (…)
>         Chunk: start = 141942960685056, len = 1073741824, type = 1,
> num_stripes = 1
>             Stripes list:
>             [ 0] Stripe: devid = 2, offset = 75414325166080
>         Chunk: start = 141944034426880, len = 33554432, type = 22,
> num_stripes = 2
>             Stripes list:
>             [ 0] Stripe: devid = 2, offset = 2034741805056
>             [ 1] Stripe: devid = 2, offset = 2034775359488
>
> All Device Extents:
>         Device extent: devid = 2, start = 1048576, len = 1073741824,
> chunk offset = 49477515739136
>         Device extent: devid = 2, start = 1074790400, len =
> 1073741824, chunk offset = 49478589480960
>         Device extent: devid = 2, start = 2148532224, len =
> 1073741824, chunk offset = 49479663222784
> (…)
>         Device extent: devid = 2, start = 75412177682432, len =
> 1073741824, chunk offset = 141940813201408
>         Device extent: devid = 2, start = 75413251424256, len =
> 1073741824, chunk offset = 141941886943232
>         Device extent: devid = 2, start = 75414325166080, len =
> 1073741824, chunk offset = 141942960685056
>
> CHECK RESULT:
> Recoverable Chunks:
>   Chunk: start = 49477515739136, len = 1073741824, type = 1, num_stripes = 1
>       Stripes list:
>       [ 0] Stripe: devid = 2, offset = 1048576
>       Block Group: start = 49477515739136, len = 1073741824, flag = 1
>       Device extent list:
>           [ 0]Device extent: devid = 2, start = 1048576, len =
> 1073741824, chunk offset = 49477515739136
>   Chunk: start = 49478589480960, len = 1073741824, type = 1, num_stripes = 1
>       Stripes list:
>       [ 0] Stripe: devid = 2, offset = 1074790400
>       Block Group: start = 49478589480960, len = 1073741824, flag = 1
>       Device extent list:
>           [ 0]Device extent: devid = 2, start = 1074790400, len =
> 1073741824, chunk offset = 49478589480960
>   Chunk: start = 49479663222784, len = 1073741824, type = 1, num_stripes = 1
>       Stripes list:
>       [ 0] Stripe: devid = 2, offset = 2148532224
>       Block Group: start = 49479663222784, len = 1073741824, flag = 1
>       Device extent list:
>           [ 0]Device extent: devid = 2, start = 2148532224, len =
> 1073741824, chunk offset = 49479663222784
> (…)
>   Chunk: start = 141085812719616, len = 1073741824, type = 1, num_stripes = 1
>       Stripes list:
>       [ 0] Stripe: devid = 2, offset = 74690623176704
>       No block group.
>       Device extent list:
>           [ 0]Device extent: devid = 2, start = 74690623176704, len =
> 1073741824, chunk offset = 141085812719616
>   Chunk: start = 141403740962816, len = 1073741824, type = 1, num_stripes = 1
>       Stripes list:
>       [ 0] Stripe: devid = 2, offset = 73223891845120
>       No block group.
>       Device extent list:
>           [ 0]Device extent: devid = 2, start = 73223891845120, len =
> 1073741824, chunk offset = 141403740962816
> Unrecoverable Chunks:
>   Chunk: start = 73953460617216, len = 1073741824, type = 1, num_stripes = 1
>       Stripes list:
>       [ 0] Stripe: devid = 2, offset = 23810225995776
>       No block group.
>       No device extent.
>   Chunk: start = 75698291081216, len = 1073741824, type = 1, num_stripes = 1
>       Stripes list:
>       [ 0] Stripe: devid = 2, offset = 25555056459776
>       No block group.
>       No device extent.
> (…)
>   Chunk: start = 139435974852608, len = 1073741824, type = 1, num_stripes = 1
>       Stripes list:
>       [ 0] Stripe: devid = 2, offset = 49594055524352
>       Block Group: start = 139435974852608, len = 1073741824, flag = 1
>       No device extent.
>   Chunk: start = 140101996773376checksum verify failed on
> 99593231630336 found 000000B6 wanted 00000000
> checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> bad tree block 124762809384960, bytenr mismatch, want=124762809384960, have=0
> ERROR: failed to read block groups: Input/output error
> open with broken chunk error
> , len = 1073741824, type = 1, num_stripes = 1
>       Stripes list:
>       [ 0] Stripe: devid = 2, offset = 58501817696256
>       Block Group: start = 140101996773376, len = 1073741824, flag = 1
>       No device extent.
>   Chunk: start = 140221215670272, len = 1073741824, type = 1, num_stripes = 1
>       Stripes list:
>       [ 0] Stripe: devid = 2, offset = 58416992092160
>       Block Group: start = 140221215670272, len = 1073741824, flag = 1
>       No device extent.
> (…)
>   Chunk: start = 141836593135616, len = 33554432, type = 22, num_stripes = 2
>       Stripes list:
>       [ 0] Stripe: devid = 2, offset = 2034741805056
>       [ 1] Stripe: devid = 2, offset = 2034775359488
>       No block group.
>       No device extent.
>   Chunk: start = 141269456125952, len = 33554432, type = 22, num_stripes = 0
>       Stripes list:
>       Block Group: start = 141269456125952, len = 33554432, flag = 22
>       No device extent.
>
> Total Chunks:           72859
>   Recoverable:          68784
>   Unrecoverable:        4075
>
> Orphan Block Groups:
>
> Orphan Device Extents:
>
> Chunk tree recovery failed
>
>
>
> So. Is this good or bad? First, there are a lot of unrecoverable
> chunks. And some of them are system chunks.
> But my biggest issue seems to be that "bad tree block" error that is
> showing up and which I'm getting any time I try to access the
> file-system. This error also seems to prevent mounting and btrfs
> check. Can I use the position given there to look up the corresponding
> chunk? If so:
>
> Block Group and Chunks around bad tree block 124762809384960:
>
>         Block Group: start = 124760809799680, len = 1073741824, flag = 1
>         Block Group: start = 124761883541504, len = 1073741824, flag = 24
>         Block Group: start = 124798390763520, len = 1073741824, flag =
> 1
>
>         Chunk: start = 124760809799680, len = 1073741824, type = 1,
> num_stripes = 1
>             Stripes list:
>             [ 0] Stripe: devid = 2, offset = 67164766732288
>         Chunk: start = 124761883541504, len = 1073741824, type = 24,
> num_stripes = 2
>             Stripes list:
>             [ 0] Stripe: devid = 2, offset = 67165840474112
>             [ 1] Stripe: devid = 2, offset = 67166914215936
>         Chunk: start = 124777989668864, len = 1073741824, type = 1,
> num_stripes = 1
>             Stripes list:
>             [ 0] Stripe: devid = 2, offset = 67183020343296
>
> The bad tree block seems to contain meta data. Can I somehow recover
> this? It seems to be DUP (2 Stripes)…
>
> > > > And you'll need to look for a snapshot name in there, find its bytenr,
> > > > and let's first see if just using that works. If it doesn't then maybe
> > > > combining it with the next most recent root tree will work.
> > >
> > > I am working backwards right now using btrfs restore -f in combination
> > > with -t. So far no success.
> >
> > Yep. I think it comes down to the chunk tree needing to be reasonable
> > first, before anything else is possible.
>
> Now I tried every possible combination with btrfs restore.
> Unfortunately I found no working solution to restore the data
> undamaged. As mentioned before the files contain "holes" of 0x00.
> These holes are exactly 4 MiB in size. Does this tell anything?
> Any ideas how to proceed from here on? Accept the data loss and try to
> repair the files as good as possible -- putting them together from
> different saves (if available) is a real grind. Or professional data
> recovery -- but wouldn't they face the same issue regarding the
> chunk-tree?
>
> Kind regards,
> Sebastian



-- 
Chris Murphy

Re: All files are damaged after btrfs restore

[Qu Wenruo]

On 2021/2/23 下午11:45, Sebastian Roller wrote:
> Hello all.
> Sorry for asking here directly, but I'm in a desperate situation and
> out of options.
> I have a 72 TB btrfs filesystem which functions as a backup drive.
> After a recent controller hardware failure while the backup was
> running, both original and backup fs were severely damaged.
>
> Kernel version is 5.7.7. btrfs-progs is (now) 5.9.
>
> At the moment I am unable to mount the btrfs filesystem.
>
> root@hikitty:~$ mount -t btrfs -o ro,recovery /dev/sdf1 /mnt/
> mount: wrong fs type, bad option, bad superblock on /dev/sdf1,
>         missing codepage or helper program, or other error
>
>         In some cases useful info is found in syslog - try
>         dmesg | tail or so.
>
> [165097.777496] BTRFS warning (device sdf1): 'recovery' is deprecated,
> use 'usebackuproot' instead
> [165097.777500] BTRFS info (device sdf1): trying to use backup root at
> mount time
> [165097.777502] BTRFS info (device sdf1): disk space caching is enabled
> [165097.777505] BTRFS info (device sdf1): has skinny extents
> [165101.721250] BTRFS error (device sdf1): bad tree block start, want
> 126718415241216 have 0

This means the tree block is completely wiped out.

And I don't believe it's just one tree block wiped out, but a much
larger range of on-disk data get wiped out.

> [165101.750951] BTRFS error (device sdf1): bad tree block start, want
> 126718415241216 have 0
> [165101.755753] BTRFS error (device sdf1): failed to verify dev
> extents against chunks: -5
> [165101.895065] BTRFS error (device sdf1): open_ctree failed
>
>
> Since I desperately need the data I ran btrfs restore.
> root@hikitty:~$ install/btrfs-progs-5.9/btrfs -v restore -i -s -m -S
> --path-regex '^/(|@(|/backup(|/home(|/.*))))$' /dev/sdf1
> /mnt/dumpo/home/
> checksum verify failed on 109911545430016 found 000000B6 wanted 00000000
> checksum verify failed on 109911545462784 found 000000B6 wanted 00000000
> checksum verify failed on 57767345897472 found 000000B6 wanted 00000000

This looks like the same problem.

> Restoring /mnt/dumpo/home/@
> Restoring /mnt/dumpo/home/@/backup
> Restoring /mnt/dumpo/home/@/backup/home
> …
> (2.1 GB of log file)
> …
> Done searching /@/backup/home
> Reached the end of the tree searching the directory
> Reached the end of the tree searching the directory
> Reached the end of the tree searching the directory
>
>
> Using that restore I was able to restore approx. 7 TB of the
> originally stored 22 TB under that directory.
> Unfortunately nearly all the files are damaged. Small text files are
> still OK. But every larger binary file is useless.
> Is there any possibility to fix the filesystem in a way, that I get
> the data less damaged?

 From the result, it looks like the on-disk data get (partially) wiped out.
I doubt if it's just simple controller failure, but more likely
something not really reaching disk or something more weird.

In short, this is really the worst case.

Thanks,
Qu


>
> So far I ran no btrfs check --repair.
>
> Since the original and the backup have been damaged any help would be
> highly appreciated.
> Thanks for your assistance.
>
> Kind regards,
> Sebastian Roller
>
> ----------------  Attachment. All outputs. -------------------
> uname -a
> Linux hikitty 5.7.7-1.el7.elrepo.x86_64 #1 SMP Wed Jul 1 11:53:16 EDT
> 2020 x86_64 x86_64 x86_64 GNU/Linux
>
>
> root@hikitty:~$ install/btrfs-progs-5.9/btrfs --version
> btrfs-progs v5.9
> (Version v5.10 fails to compile)
>
>
> root@hikitty:~$ btrfs fi show
> Label: 'history'  uuid: 56051c5f-fca6-4d54-a04e-1c1d8129fe56
>          Total devices 1 FS bytes used 68.37TiB
>          devid    2 size 72.77TiB used 68.59TiB path /dev/sdf1
>
>
> root@hikitty:~$ mount -t btrfs -o ro,recovery /dev/sdf1 /mnt/hist/
> mount: wrong fs type, bad option, bad superblock on /dev/sdf1,
>         missing codepage or helper program, or other error
>
>         In some cases useful info is found in syslog - try
>         dmesg | tail or so.
>
> [165097.777496] BTRFS warning (device sdf1): 'recovery' is deprecated,
> use 'usebackuproot' instead
> [165097.777500] BTRFS info (device sdf1): trying to use backup root at
> mount time
> [165097.777502] BTRFS info (device sdf1): disk space caching is enabled
> [165097.777505] BTRFS info (device sdf1): has skinny extents
> [165101.721250] BTRFS error (device sdf1): bad tree block start, want
> 126718415241216 have 0
> [165101.750951] BTRFS error (device sdf1): bad tree block start, want
> 126718415241216 have 0
> [165101.755753] BTRFS error (device sdf1): failed to verify dev
> extents against chunks: -5
> [165101.895065] BTRFS error (device sdf1): open_ctree failed
>
>
> root@hikitty:~$ btrfs rescue super-recover -v /dev/sdf1
> All Devices:
>          Device: id = 2, name = /dev/sdh1
>
> Before Recovering:
>          [All good supers]:
>                  device name = /dev/sdh1
>                  superblock bytenr = 65536
>
>                  device name = /dev/sdh1
>                  superblock bytenr = 67108864
>
>                  device name = /dev/sdh1
>                  superblock bytenr = 274877906944
>
>          [All bad supers]:
>
> All supers are valid, no need to recover
>
>
> root@hikitty:/mnt$ btrfs rescue chunk-recover /dev/sdf1
> Scanning: DONE in dev0
> checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
> checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
> checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
> bytenr mismatch, want=124762809384960, have=0
> open with broken chunk error
> Chunk tree recovery failed
>
> ^^ This was btrfs v4.14
>
>
> root@hikitty:~$ install/btrfs-progs-5.9/btrfs check --readonly /dev/sdi1
> Opening filesystem to check...
> checksum verify failed on 99593231630336 found 000000B6 wanted 00000000
> checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> bad tree block 124762809384960, bytenr mismatch, want=124762809384960, have=0
> ERROR: failed to read block groups: Input/output error
> ERROR: cannot open file system
>
>
> FIRST MOUNT AT BOOT TIME AFTER DESASTER
> Feb 15 08:05:11 hikitty kernel: BTRFS info (device sdf1): disk space
> caching is enabled
> Feb 15 08:05:11 hikitty kernel: BTRFS info (device sdf1): has skinny extents
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944039161856 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944039161856 (dev /dev/sdf1 sector 3974114336)
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944039165952 (dev /dev/sdf1 sector 3974114344)
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944039170048 (dev /dev/sdf1 sector 3974114352)
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944039174144 (dev /dev/sdf1 sector 3974114360)
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944037851136 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944037851136 (dev /dev/sdf1 sector 3974111776)
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944037855232 (dev /dev/sdf1 sector 3974111784)
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944037859328 (dev /dev/sdf1 sector 3974111792)
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944037863424 (dev /dev/sdf1 sector 3974111800)
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944040767488 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944040767488 (dev /dev/sdf1 sector 3974117472)
> Feb 15 08:05:12 hikitty kernel: BTRFS info (device sdf1): read error
> corrected: ino 0 off 141944040771584 (dev /dev/sdf1 sector 3974117480)
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944035147776 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944035115008 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944035131392 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944036327424 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944036278272 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944035164160 have 0
> Feb 15 08:05:12 hikitty kernel: BTRFS error (device sdf1): bad tree
> block start, want 141944036294656 have 0
> Feb 15 08:05:16 hikitty kernel: BTRFS error (device sdf1): failed to
> verify dev extents against chunks: -5
> Feb 15 08:05:16 hikitty kernel: BTRFS error (device sdf1): open_ctree failed
>

Re: All files are damaged after btrfs restore

[Dāvis Mosāns]

otrd., 2021. g. 23. febr., plkst. 17:51 — lietotājs Sebastian Roller
(<sebastian.roller@gmail.com>) rakstīja:
>
[...]
>
> root@hikitty:~$ install/btrfs-progs-5.9/btrfs check --readonly /dev/sdi1
> Opening filesystem to check...
> checksum verify failed on 99593231630336 found 000000B6 wanted 00000000
> checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> bad tree block 124762809384960, bytenr mismatch, want=124762809384960, have=0
> ERROR: failed to read block groups: Input/output error
> ERROR: cannot open file system

One possible reason could be that extent tree is corrupted. Right now
I'm dealing with such filesystem.
I wrote a patch that allows to read-only mount such filesystem
(assuming there's no other problems).

Currently none of btrfs tools work when extent tree is corrupted, but
for `btrfs check` this patch would allow it to go further. Only for
data recovery this isn't really that useful.

--- a/check/main.c
+++ b/check/main.c
@@ -10197,7 +10197,7 @@ static int cmd_check(const struct cmd_struct
*cmd, int argc, char **argv)
int qgroup_report = 0;
int qgroups_repaired = 0;
int qgroup_verify_ret;
-       unsigned ctree_flags = OPEN_CTREE_EXCLUSIVE;
+       unsigned ctree_flags = OPEN_CTREE_EXCLUSIVE |
OPEN_CTREE_NO_BLOCK_GROUPS;
int force = 0;

while(1) {

Re: All files are damaged after btrfs restore

[Chris Murphy]

On Tue, Mar 16, 2021 at 7:39 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > Using that restore I was able to restore approx. 7 TB of the
> > originally stored 22 TB under that directory.
> > Unfortunately nearly all the files are damaged. Small text files are
> > still OK. But every larger binary file is useless.
> > Is there any possibility to fix the filesystem in a way, that I get
> > the data less damaged?
>
>  From the result, it looks like the on-disk data get (partially) wiped out.
> I doubt if it's just simple controller failure, but more likely
> something not really reaching disk or something more weird.

Hey Qu, thanks for the reply.

So it's not clear until further downthread that it's bcache in
writeback mode with an SSD that failed. And I've probably
underestimated the significance of how much data (in this case both
Btrfs metadata and user data) and for how long it can stay *only* on
the SSD with this policy.

https://bcache.evilpiepirate.org/ says it straight up, if using
writeback, it is not at all safe for the cache and backing devices to
be separated. If the cache device fails, everything on it is gone. By
my reading, for example, if the writeback percent is 50%, and the
cache device is 128G, at any given time 64G is *only* on the SSD.
There's no idle time flushing to the backing device that eventually
makes the backing device possibly a self sufficient storage device on
its own, it always needs the cache device.


-- 
Chris Murphy

Re: All files are damaged after btrfs restore

[Sebastian Roller]

Am Mi., 17. März 2021 um 03:59 Uhr schrieb Chris Murphy
<lists@colorremedies.com>:
>
> On Tue, Mar 16, 2021 at 7:39 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > > Using that restore I was able to restore approx. 7 TB of the
> > > originally stored 22 TB under that directory.
> > > Unfortunately nearly all the files are damaged. Small text files are
> > > still OK. But every larger binary file is useless.
> > > Is there any possibility to fix the filesystem in a way, that I get
> > > the data less damaged?
> >
> >  From the result, it looks like the on-disk data get (partially) wiped out.
> > I doubt if it's just simple controller failure, but more likely
> > something not really reaching disk or something more weird.
>
> Hey Qu, thanks for the reply.
>
> So it's not clear until further downthread that it's bcache in
> writeback mode with an SSD that failed. And I've probably
> underestimated the significance of how much data (in this case both
> Btrfs metadata and user data) and for how long it can stay *only* on
> the SSD with this policy.

Sorry Chris. I might have expressed this wrongly. But the
btrfs-filesystem was never on bcache. On bcache was a xfs-filesystem
that I backed up (rsynced) to the btrfs-filesystem when everything
went wrong. And I quickly gave up hope for that xfs afterwards, due to
the (lost) cached data of the directory-structure. That's why I'm
focusing on getting the backup on the btrfs-filesystem back.
But the possibility that some data is really wiped out or as Qu said,
that something is not reaching the disk gives me a direction to
investigate further. Maybe the raid-enclosure or the FC got damaged (I
ruled that out in the beginning). Eventually a failed raid-rebuild or
so. That would explain why so much data is missing.
Thank you.

Sebastian

Re: All files are damaged after btrfs restore

[Sebastian Roller]

Am Mi., 17. März 2021 um 02:54 Uhr schrieb Dāvis Mosāns <davispuh@gmail.com>:

> > root@hikitty:~$ install/btrfs-progs-5.9/btrfs check --readonly /dev/sdi1
> > Opening filesystem to check...
> > checksum verify failed on 99593231630336 found 000000B6 wanted 00000000
> > checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> > checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> > checksum verify failed on 124762809384960 found 000000B6 wanted 00000000
> > bad tree block 124762809384960, bytenr mismatch, want=124762809384960, have=0
> > ERROR: failed to read block groups: Input/output error
> > ERROR: cannot open file system
>
> One possible reason could be that extent tree is corrupted. Right now
> I'm dealing with such filesystem.
> I wrote a patch that allows to read-only mount such filesystem
> (assuming there's no other problems).
>
> Currently none of btrfs tools work when extent tree is corrupted, but
> for `btrfs check` this patch would allow it to go further. Only for
> data recovery this isn't really that useful.
>
> --- a/check/main.c
> +++ b/check/main.c
> @@ -10197,7 +10197,7 @@ static int cmd_check(const struct cmd_struct
> *cmd, int argc, char **argv)
> int qgroup_report = 0;
> int qgroups_repaired = 0;
> int qgroup_verify_ret;
> -       unsigned ctree_flags = OPEN_CTREE_EXCLUSIVE;
> +       unsigned ctree_flags = OPEN_CTREE_EXCLUSIVE |
> OPEN_CTREE_NO_BLOCK_GROUPS;
> int force = 0;
>
> while(1) {

The patch works fine. Btrfs check is now running and I'm getting "bad
tree block  … have=0" for a lot of blocks. It supports the point of
Chris and Qu that some parts of the filesystem are missing completely.
I will have to look again into the hardware.
Thank you.

root@hikitty:~/install/btrfs-progs-5.11-patch$ ./btrfs check
--readonly /dev/sdi1
Opening filesystem to check...
Checking filesystem on /dev/sdi1
UUID: 56051c5f-fca6-4d54-a04e-1c1d8129fe56
[1/7] checking root items
checksum verify failed on 99593231630336 found 000000B6 wanted 00000000
checksum verify failed on 93744341237760 found 000000B6 wanted 00000000
checksum verify failed on 134571939774464 found 000000B6 wanted 00000000
checksum verify failed on 134571939774464 found 000000B6 wanted 00000000
checksum verify failed on 134571939774464 found 000000B6 wanted 00000000
bad tree block 134571939774464, bytenr mismatch, want=134571939774464, have=0
ERROR: failed to repair root items: Input/output error
[2/7] checking extents
checksum verify failed on 114674562678784 found 000000B6 wanted 00000000
checksum verify failed on 57137668423680 found 000000B6 wanted 00000000
checksum verify failed on 113640702476288 found 000000B6 wanted 00000000
checksum verify failed on 113640702476288 found 000000B6 wanted 00000000
checksum verify failed on 113640702476288 found 000000B6 wanted 00000000
bad tree block 113640702476288, bytenr mismatch, want=113640702476288, have=0
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
checksum verify failed on 121433146130432 found 000000B6 wanted 00000000
checksum verify failed on 122593932312576 found 000000B6 wanted 00000000
checksum verify failed on 136599398825984 found 000000B6 wanted 00000000
checksum verify failed on 101807915597824 found 000000B6 wanted 00000000
checksum verify failed on 57164715065344 found 000000B6 wanted 00000000
checksum verify failed on 96553819602944 found 000000B6 wanted 00000000
checksum verify failed on 117685684518912 found 000000B6 wanted 00000000
checksum verify failed on 94429572694016 found 000000B6 wanted 00000000
checksum verify failed on 103132624437248 found 000000B6 wanted 00000000
checksum verify failed on 136601755516928 found 000000B6 wanted 00000000
checksum verify failed on 57973615050752 found 000000B6 wanted 00000000
checksum verify failed on 121433143312384 found 000000B6 wanted 00000000
checksum verify failed on 136602075971584 found 000000B6 wanted 00000000
checksum verify failed on 57973809496064 found 000000B6 wanted 00000000
checksum verify failed on 110464911507456 found 000000B6 wanted 00000000
checksum verify failed on 57602967863296 found 000000B6 wanted 00000000
checksum verify failed on 121020084862976 found 000000B6 wanted 00000000
checksum verify failed on 122343797145600 found 000000B6 wanted 00000000
checksum verify failed on 136601758367744 found 000000B6 wanted 00000000
checksum verify failed on 58255412936704 found 000000B6 wanted 00000000
checksum verify failed on 98233031737344 found 000000B6 wanted 00000000
checksum verify failed on 112202434707456 found 000000B6 wanted 00000000
checksum verify failed on 112202434707456 found 000000B6 wanted 00000000
checksum verify failed on 112202434707456 found 000000B6 wanted 00000000
bad tree block 112202434707456, bytenr mismatch, want=112202434707456, have=0
checksum verify failed on 112202478764032 found 000000B6 wanted 00000000
checksum verify failed on 112202504568832 found 000000B6 wanted 00000000
checksum verify failed on 99302161956864 found 000000B6 wanted 00000000
checksum verify failed on 57229417267200 found 000000B6 wanted 00000000
checksum verify failed on 123018565664768 found 000000B6 wanted 00000000
checksum verify failed on 123018565664768 found 000000B6 wanted 00000000
checksum verify failed on 123018565664768 found 000000B6 wanted 00000000
bad tree block 123018565664768, bytenr mismatch, want=123018565664768, have=0
(…)
(killed the run after some minutes and a lot more of these errors)

--
Sebastian