BTRFS device unmountable after system freeze

BTRFS device unmountable after system freeze

[Pet Eren]

Hi there!
 
I have been using BTRFS on an external hdd (WD Elements 8TB, Single mode, data, metadata and system in DUP). The device is LUKS encrypted.
After my Ubuntu 18.04 freezed, I can unlock LUKS, but I am not able to mount the BTRFS filesystem.

- uname -a = 5.8.6-1-MANJARO
- btrfs --version = btrfs-progs v5.7
- btrfs fi show = Label: 'Elements'  uuid: 62a62962-2ad6-45db-a3ad-77d7f64983e8
                          Total devices 1 FS bytes used 2.81TiB
                          devid    1 size 7.28TiB used 5.77TiB path /dev/mapper/elements
- btrfs fi df /home = Unfortunatly I am not able to mount de btrfs filesystem
- dmesg:
[560852.004810] BTRFS info (device dm-4): disk space caching is enabled
[560852.004812] BTRFS info (device dm-4): has skinny extents
[560852.552878] BTRFS critical (device dm-4): corrupt leaf: block=290783232 slot=81 extent bytenr=2054053203968 len=262144 invalid generation, have 878116864 expect (0, 1475717]
[560852.552884] BTRFS error (device dm-4): block=290783232 read time tree block corruption detected
[560852.557557] BTRFS critical (device dm-4): corrupt leaf: block=290783232 slot=81 extent bytenr=2054053203968 len=262144 invalid generation, have 878116864 expect (0, 1475717]
[560852.557564] BTRFS error (device dm-4): block=290783232 read time tree block corruption detected
[560852.557605] BTRFS error (device dm-4): failed to read block groups: -5
[560852.616539] BTRFS error (device dm-4): open_ctree failed
 
I also tried to mount the device on another system (where the volume is created) with the same results:
- uname -a = 4.15.0-118-generic
- btrfs --version = btrfs-progs v4.15.1 
 

sudo btrfs check --readonly /dev/mapper/elements 
Ends like this (with many more "Error: invalid generation for extent" lines)
..
ERROR: invalid generation for extent 4552998985728, have 15046180803710188199 expect (0, 1475717]
ERROR: invalid generation for extent 4555984134144, have 6921944990350260720 expect (0, 1475717]
ERROR: invalid generation for extent 4556810252288, have 13383730893851772781 expect (0, 1475717]
ERROR: invalid generation for extent 4558174781440, have 8649067467592986678 expect (0, 1475717]
ERROR: invalid generation for extent 4558308999168, have 12953021474535714951 expect (0, 1475717]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
cache and super generation don't match, space cache will be invalidated
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 3086706528256 bytes used, error(s) found
total csum bytes: 3008753956
total tree bytes: 4968366080
total fs tree bytes: 1722761216
total extent tree bytes: 103022592
btree space waste bytes: 352037386
file data blocks allocated: 3189338202112
 referenced 3189312909312


Do you have any solutions how I can fix this problem?
 
Kind regards,
 
Peter
 

Re: BTRFS device unmountable after system freeze

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 3658 bytes --]



On 2020/10/4 下午6:44, Pet Eren wrote:
> Hi there!
>  
> I have been using BTRFS on an external hdd (WD Elements 8TB, Single mode, data, metadata and system in DUP). The device is LUKS encrypted.
> After my Ubuntu 18.04 freezed, I can unlock LUKS, but I am not able to mount the BTRFS filesystem.
> 
> - uname -a = 5.8.6-1-MANJARO
> - btrfs --version = btrfs-progs v5.7
> - btrfs fi show = Label: 'Elements'  uuid: 62a62962-2ad6-45db-a3ad-77d7f64983e8
>                           Total devices 1 FS bytes used 2.81TiB
>                           devid    1 size 7.28TiB used 5.77TiB path /dev/mapper/elements
> - btrfs fi df /home = Unfortunatly I am not able to mount de btrfs filesystem
> - dmesg:
> [560852.004810] BTRFS info (device dm-4): disk space caching is enabled
> [560852.004812] BTRFS info (device dm-4): has skinny extents
> [560852.552878] BTRFS critical (device dm-4): corrupt leaf: block=290783232 slot=81 extent bytenr=2054053203968 len=262144 invalid generation, have 878116864 expect (0, 1475717]
> [560852.552884] BTRFS error (device dm-4): block=290783232 read time tree block corruption detected
> [560852.557557] BTRFS critical (device dm-4): corrupt leaf: block=290783232 slot=81 extent bytenr=2054053203968 len=262144 invalid generation, have 878116864 expect (0, 1475717]
> [560852.557564] BTRFS error (device dm-4): block=290783232 read time tree block corruption detected
> [560852.557605] BTRFS error (device dm-4): failed to read block groups: -5
> [560852.616539] BTRFS error (device dm-4): open_ctree failed
>  
> I also tried to mount the device on another system (where the volume is created) with the same results:
> - uname -a = 4.15.0-118-generic
> - btrfs --version = btrfs-progs v4.15.1 
>  
> 
> sudo btrfs check --readonly /dev/mapper/elements 
> Ends like this (with many more "Error: invalid generation for extent" lines)
> ..
> ERROR: invalid generation for extent 4552998985728, have 15046180803710188199 expect (0, 1475717]
> ERROR: invalid generation for extent 4555984134144, have 6921944990350260720 expect (0, 1475717]
> ERROR: invalid generation for extent 4556810252288, have 13383730893851772781 expect (0, 1475717]
> ERROR: invalid generation for extent 4558174781440, have 8649067467592986678 expect (0, 1475717]
> ERROR: invalid generation for extent 4558308999168, have 12953021474535714951 expect (0, 1475717]

Only extent tree corruptions, not a big problem.

Nothing of your fs is lost, just kernel is too sensitive to any possible
corrupted metadata, thus rejecting it completely, to avoid further problems.

In fact, if you go several kernel version backward, you can mount the fs
without problem.

> ERROR: errors found in extent allocation tree or chunk allocation
> [3/7] checking free space cache
> cache and super generation don't match, space cache will be invalidated
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 3086706528256 bytes used, error(s) found
> total csum bytes: 3008753956
> total tree bytes: 4968366080
> total fs tree bytes: 1722761216
> total extent tree bytes: 103022592
> btree space waste bytes: 352037386
> file data blocks allocated: 3189338202112
>  referenced 3189312909312
> 
> 
> Do you have any solutions how I can fix this problem?

You can try this branch with "btrfs check --repair":

https://github.com/adam900710/btrfs-progs/tree/extent_gen_repair

Thanks,
Qu
>  
> Kind regards,
>  
> Peter
>  
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: BTRFS device unmountable after system freeze

[Chris Murphy]

On Sun, Oct 4, 2020 at 6:12 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/10/4 下午6:44, Pet Eren wrote:
> > Hi there!
> >
> > I have been using BTRFS on an external hdd (WD Elements 8TB, Single mode, data, metadata and system in DUP). The device is LUKS encrypted.
> > After my Ubuntu 18.04 freezed, I can unlock LUKS, but I am not able to mount the BTRFS filesystem.
> >
> > - uname -a = 5.8.6-1-MANJARO
> > - btrfs --version = btrfs-progs v5.7
> > - btrfs fi show = Label: 'Elements'  uuid: 62a62962-2ad6-45db-a3ad-77d7f64983e8
> >                           Total devices 1 FS bytes used 2.81TiB
> >                           devid    1 size 7.28TiB used 5.77TiB path /dev/mapper/elements
> > - btrfs fi df /home = Unfortunatly I am not able to mount de btrfs filesystem
> > - dmesg:
> > [560852.004810] BTRFS info (device dm-4): disk space caching is enabled
> > [560852.004812] BTRFS info (device dm-4): has skinny extents
> > [560852.552878] BTRFS critical (device dm-4): corrupt leaf: block=290783232 slot=81 extent bytenr=2054053203968 len=262144 invalid generation, have 878116864 expect (0, 1475717]
> > [560852.552884] BTRFS error (device dm-4): block=290783232 read time tree block corruption detected
> > [560852.557557] BTRFS critical (device dm-4): corrupt leaf: block=290783232 slot=81 extent bytenr=2054053203968 len=262144 invalid generation, have 878116864 expect (0, 1475717]
> > [560852.557564] BTRFS error (device dm-4): block=290783232 read time tree block corruption detected
> > [560852.557605] BTRFS error (device dm-4): failed to read block groups: -5
> > [560852.616539] BTRFS error (device dm-4): open_ctree failed
> >
> > I also tried to mount the device on another system (where the volume is created) with the same results:
> > - uname -a = 4.15.0-118-generic
> > - btrfs --version = btrfs-progs v4.15.1
> >
> >
> > sudo btrfs check --readonly /dev/mapper/elements
> > Ends like this (with many more "Error: invalid generation for extent" lines)
> > ..
> > ERROR: invalid generation for extent 4552998985728, have 15046180803710188199 expect (0, 1475717]
> > ERROR: invalid generation for extent 4555984134144, have 6921944990350260720 expect (0, 1475717]
> > ERROR: invalid generation for extent 4556810252288, have 13383730893851772781 expect (0, 1475717]
> > ERROR: invalid generation for extent 4558174781440, have 8649067467592986678 expect (0, 1475717]
> > ERROR: invalid generation for extent 4558308999168, have 12953021474535714951 expect (0, 1475717]
>
> Only extent tree corruptions, not a big problem.
>
> Nothing of your fs is lost, just kernel is too sensitive to any possible
> corrupted metadata, thus rejecting it completely, to avoid further problems.
>
> In fact, if you go several kernel version backward, you can mount the fs
> without problem.

Well he tried with 4.15 and got the same messages. Would '-o
usebackuproot' work here? Or is it corruption older than just this one
crash?



-- 
Chris Murphy

Re: BTRFS device unmountable after system freeze

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 3249 bytes --]



On 2020/10/5 下午2:01, Chris Murphy wrote:
> On Sun, Oct 4, 2020 at 6:12 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2020/10/4 下午6:44, Pet Eren wrote:
>>> Hi there!
>>>
>>> I have been using BTRFS on an external hdd (WD Elements 8TB, Single mode, data, metadata and system in DUP). The device is LUKS encrypted.
>>> After my Ubuntu 18.04 freezed, I can unlock LUKS, but I am not able to mount the BTRFS filesystem.
>>>
>>> - uname -a = 5.8.6-1-MANJARO
>>> - btrfs --version = btrfs-progs v5.7
>>> - btrfs fi show = Label: 'Elements'  uuid: 62a62962-2ad6-45db-a3ad-77d7f64983e8
>>>                           Total devices 1 FS bytes used 2.81TiB
>>>                           devid    1 size 7.28TiB used 5.77TiB path /dev/mapper/elements
>>> - btrfs fi df /home = Unfortunatly I am not able to mount de btrfs filesystem
>>> - dmesg:
>>> [560852.004810] BTRFS info (device dm-4): disk space caching is enabled
>>> [560852.004812] BTRFS info (device dm-4): has skinny extents
>>> [560852.552878] BTRFS critical (device dm-4): corrupt leaf: block=290783232 slot=81 extent bytenr=2054053203968 len=262144 invalid generation, have 878116864 expect (0, 1475717]
>>> [560852.552884] BTRFS error (device dm-4): block=290783232 read time tree block corruption detected
>>> [560852.557557] BTRFS critical (device dm-4): corrupt leaf: block=290783232 slot=81 extent bytenr=2054053203968 len=262144 invalid generation, have 878116864 expect (0, 1475717]
>>> [560852.557564] BTRFS error (device dm-4): block=290783232 read time tree block corruption detected
>>> [560852.557605] BTRFS error (device dm-4): failed to read block groups: -5
>>> [560852.616539] BTRFS error (device dm-4): open_ctree failed
>>>
>>> I also tried to mount the device on another system (where the volume is created) with the same results:
>>> - uname -a = 4.15.0-118-generic
>>> - btrfs --version = btrfs-progs v4.15.1
>>>
>>>
>>> sudo btrfs check --readonly /dev/mapper/elements
>>> Ends like this (with many more "Error: invalid generation for extent" lines)
>>> ..
>>> ERROR: invalid generation for extent 4552998985728, have 15046180803710188199 expect (0, 1475717]
>>> ERROR: invalid generation for extent 4555984134144, have 6921944990350260720 expect (0, 1475717]
>>> ERROR: invalid generation for extent 4556810252288, have 13383730893851772781 expect (0, 1475717]
>>> ERROR: invalid generation for extent 4558174781440, have 8649067467592986678 expect (0, 1475717]
>>> ERROR: invalid generation for extent 4558308999168, have 12953021474535714951 expect (0, 1475717]
>>
>> Only extent tree corruptions, not a big problem.
>>
>> Nothing of your fs is lost, just kernel is too sensitive to any possible
>> corrupted metadata, thus rejecting it completely, to avoid further problems.
>>
>> In fact, if you go several kernel version backward, you can mount the fs
>> without problem.
> 
> Well he tried with 4.15 and got the same messages. Would '-o
> usebackuproot' work here? Or is it corruption older than just this one
> crash?
> 

v4.15 is heavily backported, even for the stable branch.

We need to go several releases back to the v4.15 kernel which doesn't
has such check.

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 484 bytes --]