Broken Filesystem

Broken Filesystem

[Hendrik Friedel]

Hello,

I am helping someone here 
https://forum.openmediavault.org/index.php/Thread/29290-Harddrive-Failure-and-Data-Recovery/?postID=226502#post226502 
  to recover his data.
He is new to linux.

Two of his drives have a hardware problem.
btrfs filesystem show /dev/sda
Label: 'sdadisk1' uuid: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
Total devices 1 FS bytes used 128.00KiB
devid 1 size 931.51GiB used 4.10GiB path /dev/sda

The 4.1GiB are way less than what was used.


We tried to mount with mount -t btrfs -o 
recovery,nospace_cache,clear_cache

[Sat Jan 18 11:40:29 2020] BTRFS warning (device sda): 'recovery' is 
deprecated, use 'usebackuproot' instead
[Sat Jan 18 11:40:29 2020] BTRFS info (device sda): trying to use backup 
root at mount time
[Sat Jan 18 11:40:29 2020] BTRFS info (device sda): disabling disk space 
caching
[Sat Jan 18 11:40:29 2020] BTRFS info (device sda): force clearing of 
disk cache
[Sun Jan 19 11:58:24 2020] BTRFS warning (device sda): 'recovery' is 
deprecated, use 'usebackuproot' instead
[Sun Jan 19 11:58:24 2020] BTRFS info (device sda): trying to use backup 
root at mount time
[Sun Jan 19 11:58:24 2020] BTRFS info (device sda): disabling disk space 
caching
[Sun Jan 19 11:58:24 2020] BTRFS info (device sda): force clearing of 
disk cache


The mountpoint does not show any data when mounted

Scrub did not help:
btrfs scrub start /dev/sda
scrub started on /dev/sda, fsid fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16 
(pid=19881)

btrfs scrub status /dev/sda
scrub status for fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
scrub started at Sun Jan 19 12:03:35 2020 and finished after 00:00:00
total bytes scrubbed: 256.00KiB with 0 errors


btrfs check /dev/sda
Checking filesystem on /dev/sda
UUID: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 131072 bytes used err is 0
total csum bytes: 0
total tree bytes: 131072
total fs tree bytes: 32768
total extent tree bytes: 16384
btree space waste bytes: 123986
file data blocks allocated: 0
referenced 0


Also btrfs restore -i -v /dev/sda /srv/dev-disk-by-label-NewDrive2 | tee 
/restorelog.txt did not help:
It came immediately back with 'Reached the end of the tree searching the 
directory'


btrfs-find-root /dev/sda
Superblock thinks the generation is 8
Superblock thinks the level is 0
It did not finish even in 54 hours

I am out of ideas. Can you give further advice?

Regards,
Hendrik

Re: Broken Filesystem

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 3038 bytes --]



On 2020/1/25 下午7:34, Hendrik Friedel wrote:
> Hello,
> 
> I am helping someone here
> https://forum.openmediavault.org/index.php/Thread/29290-Harddrive-Failure-and-Data-Recovery/?postID=226502#post226502
>  to recover his data.
> He is new to linux.
> 
> Two of his drives have a hardware problem.
> btrfs filesystem show /dev/sda
> Label: 'sdadisk1' uuid: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
> Total devices 1 FS bytes used 128.00KiB
> devid 1 size 931.51GiB used 4.10GiB path /dev/sda
> 
> The 4.1GiB are way less than what was used.
> 
> 
> We tried to mount with mount -t btrfs -o recovery,nospace_cache,clear_cache
> 
> [Sat Jan 18 11:40:29 2020] BTRFS warning (device sda): 'recovery' is
> deprecated, use 'usebackuproot' instead
> [Sat Jan 18 11:40:29 2020] BTRFS info (device sda): trying to use backup
> root at mount time
> [Sat Jan 18 11:40:29 2020] BTRFS info (device sda): disabling disk space
> caching
> [Sat Jan 18 11:40:29 2020] BTRFS info (device sda): force clearing of
> disk cache
> [Sun Jan 19 11:58:24 2020] BTRFS warning (device sda): 'recovery' is
> deprecated, use 'usebackuproot' instead
> [Sun Jan 19 11:58:24 2020] BTRFS info (device sda): trying to use backup
> root at mount time
> [Sun Jan 19 11:58:24 2020] BTRFS info (device sda): disabling disk space
> caching
> [Sun Jan 19 11:58:24 2020] BTRFS info (device sda): force clearing of
> disk cache
> 
> 
> The mountpoint does not show any data when mounted
> 
> Scrub did not help:
> btrfs scrub start /dev/sda
> scrub started on /dev/sda, fsid fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
> (pid=19881)
> 
> btrfs scrub status /dev/sda
> scrub status for fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
> scrub started at Sun Jan 19 12:03:35 2020 and finished after 00:00:00
> total bytes scrubbed: 256.00KiB with 0 errors
> 
> 
> btrfs check /dev/sda
> Checking filesystem on /dev/sda
> UUID: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
> checking extents
> checking free space cache
> cache and super generation don't match, space cache will be invalidated
> checking fs roots
> checking csums
> checking root refs
> found 131072 bytes used err is 0
> total csum bytes: 0
> total tree bytes: 131072
> total fs tree bytes: 32768
> total extent tree bytes: 16384
> btree space waste bytes: 123986
> file data blocks allocated: 0
> referenced 0

Your fs is completely fine. I didn't see anything wrong from your `btrfs
check` result nor your kernel messages.

> 
> 
> Also btrfs restore -i -v /dev/sda /srv/dev-disk-by-label-NewDrive2 | tee
> /restorelog.txt did not help:
> It came immediately back with 'Reached the end of the tree searching the
> directory'
> 
> 
> btrfs-find-root /dev/sda
> Superblock thinks the generation is 8
> Superblock thinks the level is 0
> It did not finish even in 54 hours
> 
> I am out of ideas. Can you give further advice?

Since your fs is OK, what's wrong?

Maybe just mounted wrong subvolume?

Thanks,
Qu

> 
> Regards,
> Hendrik
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Broken Filesystem

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 4155 bytes --]



On 2020/1/25 下午9:32, Hendrik Friedel wrote:
> Hello,
> 
> thanks for your reply.
> The problem is, that no data is shown after mounting. Furthermore 
> devid 1 size 931.51GiB used 4.10GiB path /dev/sda
> The disk was almost full. Now 4.1GiB seem to be used only.

Then it looks like related subvolumes get deleted.

Without some exotic method like `btrfs-find-root` there is no way to
locate deleted subvolume data.

Thanks,
Qu

> 
> Greetings,
> Hendrik
> 
> ------ Originalnachricht ------
> Von: "Qu Wenruo" <quwenruo.btrfs@gmx.com <mailto:quwenruo.btrfs@gmx.com>>
> An: "Hendrik Friedel" <hendrik@friedels.name
> <mailto:hendrik@friedels.name>>; "Btrfs BTRFS"
> <linux-btrfs@vger.kernel.org <mailto:linux-btrfs@vger.kernel.org>>
> Gesendet: 25.01.2020 13:20:14
> Betreff: Re: Broken Filesystem
> 
>>  
>>  
>> On 2020/1/25 下午7:34, Hendrik Friedel wrote:
>>> Hello,
>>>  
>>> I am helping someone here
>>> https://forum.openmediavault.org/index.php/Thread/29290-Harddrive-Failure-and-Data-Recovery/?postID=226502#post226502
>>>  to recover his data.
>>> He is new to linux.
>>>  
>>> Two of his drives have a hardware problem.
>>> btrfs filesystem show /dev/sda
>>> Label: 'sdadisk1' uuid: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
>>> Total devices 1 FS bytes used 128.00KiB
>>> devid 1 size 931.51GiB used 4.10GiB path /dev/sda
>>>  
>>> The 4.1GiB are way less than what was used.
>>>  
>>>  
>>> We tried to mount with mount -t btrfs -o
>>> recovery,nospace_cache,clear_cache
>>>  
>>> [Sat Jan 18 11:40:29 2020] BTRFS warning (device sda): 'recovery' is
>>> deprecated, use 'usebackuproot' instead
>>> [Sat Jan 18 11:40:29 2020] BTRFS info (device sda): trying to use backup
>>> root at mount time
>>> [Sat Jan 18 11:40:29 2020] BTRFS info (device sda): disabling disk space
>>> caching
>>> [Sat Jan 18 11:40:29 2020] BTRFS info (device sda): force clearing of
>>> disk cache
>>> [Sun Jan 19 11:58:24 2020] BTRFS warning (device sda): 'recovery' is
>>> deprecated, use 'usebackuproot' instead
>>> [Sun Jan 19 11:58:24 2020] BTRFS info (device sda): trying to use backup
>>> root at mount time
>>> [Sun Jan 19 11:58:24 2020] BTRFS info (device sda): disabling disk space
>>> caching
>>> [Sun Jan 19 11:58:24 2020] BTRFS info (device sda): force clearing of
>>> disk cache
>>>  
>>>  
>>> The mountpoint does not show any data when mounted
>>>  
>>> Scrub did not help:
>>> btrfs scrub start /dev/sda
>>> scrub started on /dev/sda, fsid fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
>>> (pid=19881)
>>>  
>>> btrfs scrub status /dev/sda
>>> scrub status for fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
>>> scrub started at Sun Jan 19 12:03:35 2020 and finished after 00:00:00
>>> total bytes scrubbed: 256.00KiB with 0 errors
>>>  
>>>  
>>> btrfs check /dev/sda
>>> Checking filesystem on /dev/sda
>>> UUID: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
>>> checking extents
>>> checking free space cache
>>> cache and super generation don't match, space cache will be invalidated
>>> checking fs roots
>>> checking csums
>>> checking root refs
>>> found 131072 bytes used err is 0
>>> total csum bytes: 0
>>> total tree bytes: 131072
>>> total fs tree bytes: 32768
>>> total extent tree bytes: 16384
>>> btree space waste bytes: 123986
>>> file data blocks allocated: 0
>>> referenced 0
>>  
>> Your fs is completely fine. I didn't see anything wrong from your `btrfs
>> check` result nor your kernel messages.
>>  
>>>  
>>>  
>>> Also btrfs restore -i -v /dev/sda /srv/dev-disk-by-label-NewDrive2 | tee
>>> /restorelog.txt did not help:
>>> It came immediately back with 'Reached the end of the tree searching the
>>> directory'
>>>  
>>>  
>>> btrfs-find-root /dev/sda
>>> Superblock thinks the generation is 8
>>> Superblock thinks the level is 0
>>> It did not finish even in 54 hours
>>>  
>>> I am out of ideas. Can you give further advice?
>>  
>> Since your fs is OK, what's wrong?
>>  
>> Maybe just mounted wrong subvolume?
>>  
>> Thanks,
>> Qu
>>  
>>>  
>>> Regards,
>>> Hendrik
>>>  
>>  


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Broken Filesystem

[Hugo Mills]

On Sat, Jan 25, 2020 at 11:34:10AM +0000, Hendrik Friedel wrote:
> Hello,
> 
> I am helping someone here https://forum.openmediavault.org/index.php/Thread/29290-Harddrive-Failure-and-Data-Recovery/?postID=226502#post226502
> to recover his data.
> He is new to linux.
> 
> Two of his drives have a hardware problem.
> btrfs filesystem show /dev/sda
> Label: 'sdadisk1' uuid: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
> Total devices 1 FS bytes used 128.00KiB
> devid 1 size 931.51GiB used 4.10GiB path /dev/sda
> 
> The 4.1GiB are way less than what was used.
> 
> 
> We tried to mount with mount -t btrfs -o recovery,nospace_cache,clear_cache
> 
> [Sat Jan 18 11:40:29 2020] BTRFS warning (device sda): 'recovery' is
> deprecated, use 'usebackuproot' instead
> [Sat Jan 18 11:40:29 2020] BTRFS info (device sda): trying to use backup
> root at mount time
> [Sat Jan 18 11:40:29 2020] BTRFS info (device sda): disabling disk space
> caching
> [Sat Jan 18 11:40:29 2020] BTRFS info (device sda): force clearing of disk
> cache
> [Sun Jan 19 11:58:24 2020] BTRFS warning (device sda): 'recovery' is
> deprecated, use 'usebackuproot' instead
> [Sun Jan 19 11:58:24 2020] BTRFS info (device sda): trying to use backup
> root at mount time
> [Sun Jan 19 11:58:24 2020] BTRFS info (device sda): disabling disk space
> caching
> [Sun Jan 19 11:58:24 2020] BTRFS info (device sda): force clearing of disk
> cache
> 
> 
> The mountpoint does not show any data when mounted

   After you mount the FS, do you see the mounted filesystem in the
output of "mount"? If not, then you've probably got systemd thinking
that the FS shouldn't be mounted there, and unmounting it again
immediately.

   If you do see the FS in the output of "mount", then there's
something else going on, like files or subvolumes have been deleted,
or you're not mounting the subvol you think you are -- try mounting
with -o subvolid=0 to see the whole FS.

   Hugo.

> Scrub did not help:
> btrfs scrub start /dev/sda
> scrub started on /dev/sda, fsid fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
> (pid=19881)
> 
> btrfs scrub status /dev/sda
> scrub status for fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
> scrub started at Sun Jan 19 12:03:35 2020 and finished after 00:00:00
> total bytes scrubbed: 256.00KiB with 0 errors
> 
> 
> btrfs check /dev/sda
> Checking filesystem on /dev/sda
> UUID: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
> checking extents
> checking free space cache
> cache and super generation don't match, space cache will be invalidated
> checking fs roots
> checking csums
> checking root refs
> found 131072 bytes used err is 0
> total csum bytes: 0
> total tree bytes: 131072
> total fs tree bytes: 32768
> total extent tree bytes: 16384
> btree space waste bytes: 123986
> file data blocks allocated: 0
> referenced 0
> 
> 
> Also btrfs restore -i -v /dev/sda /srv/dev-disk-by-label-NewDrive2 | tee
> /restorelog.txt did not help:
> It came immediately back with 'Reached the end of the tree searching the
> directory'
> 
> 
> btrfs-find-root /dev/sda
> Superblock thinks the generation is 8
> Superblock thinks the level is 0
> It did not finish even in 54 hours
> 
> I am out of ideas. Can you give further advice?
> 
> Regards,
> Hendrik
> 

-- 
Hugo Mills             | Prisoner unknown: Return to Zenda.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

Re: Broken Filesystem

[Chris Murphy]

On Sat, Jan 25, 2020 at 4:34 AM Hendrik Friedel <hendrik@friedels.name> wrote:
>
> total tree bytes: 131072

and

>
> btrfs-find-root /dev/sda
> Superblock thinks the generation is 8
> Superblock thinks the level is 0
> It did not finish even in 54 hours

I agree with the recommendations thus far. But this generation sticks
out for me, along with the total tree bytes. It suggests this device
has recently been formatted?

Anyway, I advise poking around and making no changes to anything: no
--repair, no rw mount. Only do things that are read-only or are
completely reversible. Many data loss stories are the result of making
changes, "repairs" that end up making the problem much worse, more
complicate, impossible to reverse - and then what might have been
possible to recover, no longer (practically) possible. I suggest
asking the owner to reconstruct a detail version of every single step
they took since the volume was in working order.

-- 
Chris Murphy

Re: Broken filesystem

[Roderick Johnstone]

On 19/02/2019 12:34, Qu Wenruo wrote:
> 
> 
> On 2019/2/19 下午6:24, Roderick Johnstone wrote:
>> Hi
>>
>> This is on Fedora 28:
>>
>> # uname -a
>> Linux mysystem.mydomain 4.20.7-100.fc28.x86_64 #1 SMP Wed Feb 6 19:17:09
>> UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>>
>> # btrfs --version
>> btrfs-progs v4.17.1
>>
>> #   btrfs fi show
>> Label: none  uuid: 56d0171a-440d-47ff-ad0f-f7f97df31f7b
>>          Total devices 1 FS bytes used 7.39TiB
>>          devid    1 size 9.10TiB used 7.50TiB path /dev/md2
>>
>>
>> My btrfs filesystem is in a bad state after a partial disk failure on
>> the md device (raid 6 array) the file system was on.
>>
>> One of the disks had bad blocks, but instead of being ejected from the
>> array, the array hung up.
> 
> I'm a little interested why RAID6 hung up.

I'm not sure but I think it was due to the failure more of the hardware.

> 
>> After rebooting to regain access and remove
>> the bad disk I am in the following situation:
>>
>> # mount -t btrfs -o compress-force=zlib,noatime /dev/md2 /mnt/rmj
>> mount: /mnt/rmj: wrong fs type, bad option, bad superblock on /dev/md2,
>> missing codepage or helper program, or other error.
>> # dmesg
>> ...
>>    264.527647] BTRFS info (device md2): force zlib compression, level 3
>> [  264.955360] BTRFS error (device md2): parent transid verify failed on
>> 5568287064064 wanted 254988 found 94122
> 
> It's 99% some extent tree blocks get corrupted.
> 
>> [  264.964273] BTRFS error (device md2): open_ctree failed
>>
>> I can mount and access the filesystem with the usebackuproot option:
>>
>> # mount -t btrfs -o usebackuproot,compress-force=zlib,noatime /dev/md2
>> /mnt/rmj
>> [  307.542761] BTRFS info (device md2): trying to use backup root at
>> mount time
>> [  307.542768] BTRFS info (device md2): force zlib compression, level 3
>> [  307.570897] BTRFS error (device md2): parent transid verify failed on
>> 5568287064064 wanted 254988 found 94122
>> [  307.570979] BTRFS error (device md2): parent transid verify failed on
>> 5568287064064 wanted 254988 found 94122
>> [  431.167149] BTRFS info (device md2): checking UUID tree
>>
>> But later after a umount there are these messages.
>>
>> # umount /mnt/rmj
>> 2205.778998] BTRFS error (device md2): parent transid verify failed on
>> 5568276393984 wanted 254986 found 94117
>> [ 2205.779008] BTRFS: error (device md2) in __btrfs_free_extent:6831:
>> errno=-5 IO failure
>> [ 2205.779082] BTRFS info (device md2): forced readonly
>> [ 2205.779087] BTRFS: error (device md2) in btrfs_run_delayed_refs:2978:
>> errno=-5 IO failure
>> [ 2205.779192] BTRFS warning (device md2): btrfs_uuid_scan_kthread
>> failed -30
> Of course it's extent tree corrupted.
> 
>>
>> and a subsequent mount without the userbackuproot fails in the same way
>> as before.
>>
>> I have a copy of the important directories, but would like to be able to
>> repair the filesystem if possible,
> 
> You could mostly salvage the data, either use 'usebackuproot' mount
> option + RO mount or btrfs-restore.
> 
> For full rw recovery, I don't think there is a good tool right now.
> Extent tree repair is pretty trikcy, under most case, the only method is
> --init-extent-tree, but that functionality isn't tried by many users.
> And it only makes sense if all other trees are OK.
> 
> So in short, RW recovery is near impossible.
> 
>>
>> Any advise around repairing the filesystem would be appreciated.
> 
> It's better to salvage your data first and if you like adventure, try
> --init-extent-tree.
> If not, just rebuild the array.


ok, thanks.

Roderick
> 
> Thanks,
> Qu
> 
>>
>> Thanks.
>>
>> Roderick Johnstone
> 

Re: Broken filesystem

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 3518 bytes --]



On 2019/2/19 下午6:24, Roderick Johnstone wrote:
> Hi
> 
> This is on Fedora 28:
> 
> # uname -a
> Linux mysystem.mydomain 4.20.7-100.fc28.x86_64 #1 SMP Wed Feb 6 19:17:09
> UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> 
> # btrfs --version
> btrfs-progs v4.17.1
> 
> #   btrfs fi show
> Label: none  uuid: 56d0171a-440d-47ff-ad0f-f7f97df31f7b
>         Total devices 1 FS bytes used 7.39TiB
>         devid    1 size 9.10TiB used 7.50TiB path /dev/md2
> 
> 
> My btrfs filesystem is in a bad state after a partial disk failure on
> the md device (raid 6 array) the file system was on.
> 
> One of the disks had bad blocks, but instead of being ejected from the
> array, the array hung up.

I'm a little interested why RAID6 hung up.

> After rebooting to regain access and remove
> the bad disk I am in the following situation:
> 
> # mount -t btrfs -o compress-force=zlib,noatime /dev/md2 /mnt/rmj
> mount: /mnt/rmj: wrong fs type, bad option, bad superblock on /dev/md2,
> missing codepage or helper program, or other error.
> # dmesg
> ...
>   264.527647] BTRFS info (device md2): force zlib compression, level 3
> [  264.955360] BTRFS error (device md2): parent transid verify failed on
> 5568287064064 wanted 254988 found 94122

It's 99% some extent tree blocks get corrupted.

> [  264.964273] BTRFS error (device md2): open_ctree failed
> 
> I can mount and access the filesystem with the usebackuproot option:
> 
> # mount -t btrfs -o usebackuproot,compress-force=zlib,noatime /dev/md2
> /mnt/rmj
> [  307.542761] BTRFS info (device md2): trying to use backup root at
> mount time
> [  307.542768] BTRFS info (device md2): force zlib compression, level 3
> [  307.570897] BTRFS error (device md2): parent transid verify failed on
> 5568287064064 wanted 254988 found 94122
> [  307.570979] BTRFS error (device md2): parent transid verify failed on
> 5568287064064 wanted 254988 found 94122
> [  431.167149] BTRFS info (device md2): checking UUID tree
> 
> But later after a umount there are these messages.
> 
> # umount /mnt/rmj
> 2205.778998] BTRFS error (device md2): parent transid verify failed on
> 5568276393984 wanted 254986 found 94117
> [ 2205.779008] BTRFS: error (device md2) in __btrfs_free_extent:6831:
> errno=-5 IO failure
> [ 2205.779082] BTRFS info (device md2): forced readonly
> [ 2205.779087] BTRFS: error (device md2) in btrfs_run_delayed_refs:2978:
> errno=-5 IO failure
> [ 2205.779192] BTRFS warning (device md2): btrfs_uuid_scan_kthread
> failed -30
Of course it's extent tree corrupted.

> 
> and a subsequent mount without the userbackuproot fails in the same way
> as before.
> 
> I have a copy of the important directories, but would like to be able to
> repair the filesystem if possible,

You could mostly salvage the data, either use 'usebackuproot' mount
option + RO mount or btrfs-restore.

For full rw recovery, I don't think there is a good tool right now.
Extent tree repair is pretty trikcy, under most case, the only method is
--init-extent-tree, but that functionality isn't tried by many users.
And it only makes sense if all other trees are OK.

So in short, RW recovery is near impossible.

> 
> Any advise around repairing the filesystem would be appreciated.

It's better to salvage your data first and if you like adventure, try
--init-extent-tree.
If not, just rebuild the array.

Thanks,
Qu

> 
> Thanks.
> 
> Roderick Johnstone


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Broken filesystem

[Roderick Johnstone]

Hi

This is on Fedora 28:

# uname -a
Linux mysystem.mydomain 4.20.7-100.fc28.x86_64 #1 SMP Wed Feb 6 19:17:09 
UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

# btrfs --version
btrfs-progs v4.17.1

#   btrfs fi show
Label: none  uuid: 56d0171a-440d-47ff-ad0f-f7f97df31f7b
         Total devices 1 FS bytes used 7.39TiB
         devid    1 size 9.10TiB used 7.50TiB path /dev/md2


My btrfs filesystem is in a bad state after a partial disk failure on 
the md device (raid 6 array) the file system was on.

One of the disks had bad blocks, but instead of being ejected from the 
array, the array hung up. After rebooting to regain access and remove 
the bad disk I am in the following situation:

# mount -t btrfs -o compress-force=zlib,noatime /dev/md2 /mnt/rmj
mount: /mnt/rmj: wrong fs type, bad option, bad superblock on /dev/md2, 
missing codepage or helper program, or other error.
# dmesg
...
   264.527647] BTRFS info (device md2): force zlib compression, level 3
[  264.955360] BTRFS error (device md2): parent transid verify failed on 
5568287064064 wanted 254988 found 94122
[  264.964273] BTRFS error (device md2): open_ctree failed

I can mount and access the filesystem with the usebackuproot option:

# mount -t btrfs -o usebackuproot,compress-force=zlib,noatime /dev/md2 
/mnt/rmj
[  307.542761] BTRFS info (device md2): trying to use backup root at 
mount time
[  307.542768] BTRFS info (device md2): force zlib compression, level 3
[  307.570897] BTRFS error (device md2): parent transid verify failed on 
5568287064064 wanted 254988 found 94122
[  307.570979] BTRFS error (device md2): parent transid verify failed on 
5568287064064 wanted 254988 found 94122
[  431.167149] BTRFS info (device md2): checking UUID tree

But later after a umount there are these messages.

# umount /mnt/rmj
2205.778998] BTRFS error (device md2): parent transid verify failed on 
5568276393984 wanted 254986 found 94117
[ 2205.779008] BTRFS: error (device md2) in __btrfs_free_extent:6831: 
errno=-5 IO failure
[ 2205.779082] BTRFS info (device md2): forced readonly
[ 2205.779087] BTRFS: error (device md2) in btrfs_run_delayed_refs:2978: 
errno=-5 IO failure
[ 2205.779192] BTRFS warning (device md2): btrfs_uuid_scan_kthread 
failed -30

and a subsequent mount without the userbackuproot fails in the same way 
as before.

I have a copy of the important directories, but would like to be able to 
repair the filesystem if possible,

Any advise around repairing the filesystem would be appreciated.

Thanks.

Roderick Johnstone