Corrupted btrfs (LUKS), seeking advice

Corrupted btrfs (LUKS), seeking advice

[Michael Zacherl]

Hello,
on the occasion of retrofitting a 2TB ssd for my old XPS13 9350 I decided to give btrfs w/ encryption a try (this was in June).
Now, by a dumb mistake, I have a corrupted btrfs (LUKS encrypted).
Since I can't boot from this partition anymore I'm using the distro's live system.
This partition can't be mounted.

# uname -a
Linux EndeavourOS 5.18.5-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 16 Jun 2022 20:40:45 +0000 x86_64 GNU/Linux

# btrfs --version
btrfs-progs v5.18.1

What I did so far:

# cryptsetup open /dev/nvme0n1p2 luks-test
Enter passphrase for /dev/nvme0n1p2:
[worked]
# mount -o ro,rescue=usebackuproot  /dev/mapper/luks-test /mnt
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/mapper/luks-test, missing codepage or helper program, or other error.
        dmesg(1) may have more information after failed mount system call.

      dmesg after this mount attempt:
[ 5179.422225] BTRFS info (device dm-1): flagging fs with big metadata feature
[ 5179.422248] BTRFS info (device dm-1): trying to use backup root at mount time
[ 5179.422255] BTRFS info (device dm-1): using free space tree
[ 5179.422260] BTRFS info (device dm-1): has skinny extents
[ 5179.431117] BTRFS error (device dm-1): parent transid verify failed on 334692352 wanted 14761 found 14765
[ 5179.431338] BTRFS error (device dm-1): parent transid verify failed on 334692352 wanted 14761 found 14765
[ 5179.431358] BTRFS error (device dm-1): failed to read block groups: -5
[ 5179.433358] BTRFS error (device dm-1): open_ctree failed

# btrfs check /dev/mapper/luks-test 2>&1|less -I
parent transid verify failed on 334692352 wanted 14761 found 14765
parent transid verify failed on 334692352 wanted 14761 found 14765
parent transid verify failed on 334692352 wanted 14761 found 14765
Ignoring transid failure
[1/7] checking root items
parent transid verify failed on 334643200 wanted 14761 found 14765
parent transid verify failed on 334643200 wanted 14761 found 14765
parent transid verify failed on 334643200 wanted 14761 found 14765
Ignoring transid failure
parent transid verify failed on 334659584 wanted 14761 found 14765
parent transid verify failed on 334659584 wanted 14761 found 14765
parent transid verify failed on 334659584 wanted 14761 found 14765
Ignoring transid failure
parent transid verify failed on 334430208 wanted 6728 found 14763
parent transid verify failed on 334430208 wanted 6728 found 14763
parent transid verify failed on 334430208 wanted 6728 found 14763
Ignoring transid failure
parent transid verify failed on 334675968 wanted 14761 found 14765
parent transid verify failed on 334675968 wanted 14761 found 14765
parent transid verify failed on 334675968 wanted 14761 found 14765
Ignoring transid failure
parent transid verify failed on 335216640 wanted 6728 found 14765
parent transid verify failed on 335216640 wanted 6728 found 14765
parent transid verify failed on 335216640 wanted 6728 found 14765
Ignoring transid failure
parent transid verify failed on 320847872 wanted 14323 found 14763
parent transid verify failed on 320847872 wanted 14323 found 14763
parent transid verify failed on 320847872 wanted 14323 found 14763
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=119848960 item=49 parent level=1 child bytenr=320847872 child level=1
ERROR: failed to repair root items: Input/output error
[2/7] checking extents
parent transid verify failed on 340246528 wanted 14741 found 14764
parent transid verify failed on 340246528 wanted 14741 found 14764
parent transid verify failed on 340246528 wanted 14741 found 14764
Ignoring transid failure
[... skipping many lines]
root 257 inode 1866942 errors 2001, no inode item, link count wrong
         unresolved ref dir 5719 index 9016 namelen 28 name SiteSecurityServiceState.txt filetype 1 errors 4, no inode ref
root 257 inode 1866943 errors 2001, no inode item, link count wrong
         unresolved ref dir 5719 index 9018 namelen 21 name AlternateServices.txt filetype 1 errors 4, no inode ref
root 257 inode 1866989 errors 2001, no inode item, link count wrong
         unresolved ref dir 346 index 16043 namelen 4 name user filetype 1 errors 4, no inode ref
root 257 inode 1866990 errors 2001, no inode item, link count wrong
         unresolved ref dir 1216 index 11701 namelen 46 name 0_3_1920_1080_8b47947fd8179de11b12e22fa2a454c8 filetype 1 errors 4, no inode ref
root 257 inode 1866991 errors 2001, no inode item, link count wrong
         unresolved ref dir 5720 index 6961 namelen 16 name recovery.jsonlz4 filetype 1 errors 4, no inode ref
root 257 inode 1866995 errors 2001, no inode item, link count wrong
         unresolved ref dir 6765 index 134 namelen 42 name 3647222921wleabcEoxlt-eengsairo.sqlite-wal filetype 1 errors 4, no inode ref
parent transid verify failed on 348258304 wanted 14749 found 14766
Ignoring transid failure
parent transid verify failed on 348258304 wanted 14749 found 14766
Ignoring transid failure
parent transid verify failed on 348258304 wanted 14749 found 14766
Ignoring transid failure
parent transid verify failed on 348258304 wanted 14749 found 14766
Ignoring transid failure
ERROR: errors found in fs roots
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-test
UUID: 2d1dc6b4-84ab-4c64-91a0-669b6228c516
found 83720974336 bytes used, error(s) found
total csum bytes: 50467580
total tree bytes: 266665984
total fs tree bytes: 186236928
total extent tree bytes: 17317888
btree space waste bytes: 40645922
file data blocks allocated: 87345299456
  referenced 47847440384
[less: lines 1364572-1364611/1364611 byte 90015202/90015202 (END)  (press RETURN)]

# btrfs-find-root /dev/mapper/luks-test
parent transid verify failed on 334692352 wanted 14761 found 14765
parent transid verify failed on 334692352 wanted 14761 found 14765
ERROR: failed to read block groups: Input/output error
Superblock thinks the generation is 14761
Superblock thinks the level is 0
Found tree root at 444612608 gen 14761 level 0
Well block 347160576(gen: 14768 level: 0) seems good, but generation/level doesn't match, want gen: 14761 level: 0
Well block 348192768(gen: 14766 level: 0) seems good, but generation/level doesn't match, want gen: 14761 level: 0
Well block 347865088(gen: 14765 level: 0) seems good, but generation/level doesn't match, want gen: 14761 level: 0
Well block 418840576(gen: 14758 level: 0) seems good, but generation/level doesn't match, want gen: 14761 level: 0
Well block 417120256(gen: 14749 level: 0) seems good, but generation/level doesn't match, want gen: 14761 level: 0
Well block 352256000(gen: 14743 level: 0) seems good, but generation/level doesn't match, want gen: 14761 level: 0
[end]

This is what I found out by reading.
A fix - if possible - is out of my league now and don't want to poke around and make things worse.
Any chance to get this FS at least mounted for RO?

Thanks a lot, Michael.

Re: Corrupted btrfs (LUKS), seeking advice

[Chris Murphy]

On Mon, Aug 8, 2022, at 9:06 AM, Michael Zacherl wrote:
> 
> # mount -o ro,rescue=usebackuproot  /dev/mapper/luks-test /mnt

Try

mount -o ro,rescue=all

If that works you can umount and try again adding all the rescue options except idatacsums. It's nice to have datacsum warnings (unless there's just to many errors.)

Otherwise its btrfs restore, which is tedious to use but you should still be able to get data off.

-- 
Chris Murphy

Re: Corrupted btrfs (LUKS), seeking advice

[Michael Zacherl]

On 8/8/22 16:57, Chris Murphy wrote:
> mount -o ro,rescue=all
> 
> If that works you can umount and try again adding all the rescue options except idatacsums.> It's nice to have datacsum warnings (unless there's just to many errors.)

rescue=all mounts.

dmesg from   mount -o ro,rescue=usebackuproot,rescue=nologreplay,rescue=ignorebadroots /dev/mapper/luks-test /mnt :

[20680.066631] BTRFS info (device dm-1): flagging fs with big metadata feature
[20680.066641] BTRFS info (device dm-1): trying to use backup root at mount time
[20680.066646] BTRFS info (device dm-1): disabling log replay at mount time
[20680.066650] BTRFS info (device dm-1): ignoring bad roots
[20680.066652] BTRFS info (device dm-1): using free space tree
[20680.066654] BTRFS info (device dm-1): has skinny extents
[20680.072359] BTRFS error (device dm-1): parent transid verify failed on 334692352 wanted 14761 found 14765
[20680.072571] BTRFS error (device dm-1): parent transid verify failed on 334692352 wanted 14761 found 14765
[20680.073296] BTRFS info (device dm-1): enabling ssd optimizations

A brief look shows I can access data!

When also omitting nologreplay the FS wouldn't mount and dmesg shows

[20837.557120] BTRFS info (device dm-1): flagging fs with big metadata feature
[20837.557138] BTRFS info (device dm-1): trying to use backup root at mount time
[20837.557148] BTRFS info (device dm-1): ignoring bad roots
[20837.557152] BTRFS info (device dm-1): using free space tree
[20837.557156] BTRFS info (device dm-1): has skinny extents
[20837.567354] BTRFS error (device dm-1): parent transid verify failed on 334692352 wanted 14761 found 14765
[20837.567809] BTRFS error (device dm-1): parent transid verify failed on 334692352 wanted 14761 found 14765
[20837.569387] BTRFS info (device dm-1): enabling ssd optimizationsquite understans
[20837.569403] BTRFS info (device dm-1): start tree-log replay
[20837.637057] BTRFS error (device dm-1): parent transid verify failed on 337412096 wanted 5492 found 14764
[20837.637223] BTRFS error (device dm-1): parent transid verify failed on 337412096 wanted 5492 found 14764
[20837.656541] BTRFS error (device dm-1): parent transid verify failed on 334643200 wanted 14761 found 14765
[20837.656670] BTRFS error (device dm-1): parent transid verify failed on 334643200 wanted 14761 found 14765
[20837.656676] BTRFS: error (device dm-1: state A) in btrfs_run_delayed_refs:2151: errno=-5 IO failure
[20837.656682] BTRFS: error (device dm-1: state EA) in btrfs_replay_log:2567: errno=-5 IO failure (Failed to recover log tree)
[20837.675238] BTRFS error (device dm-1: state EA): open_ctree failed

Is this FS repairable to a usable state?

Thanks a lot, Michael.

Re: Corrupted btrfs (LUKS), seeking advice

[Qu Wenruo]

On 2022/8/8 21:06, Michael Zacherl wrote:
> Hello,
> on the occasion of retrofitting a 2TB ssd for my old XPS13 9350 I 
> decided to give btrfs w/ encryption a try (this was in June).
> Now, by a dumb mistake, I have a corrupted btrfs (LUKS encrypted).
> Since I can't boot from this partition anymore I'm using the distro's 
> live system.
> This partition can't be mounted.
> 
> # uname -a
> Linux EndeavourOS 5.18.5-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 16 Jun 2022 
> 20:40:45 +0000 x86_64 GNU/Linux
> 
> # btrfs --version
> btrfs-progs v5.18.1
> 
> What I did so far:
> 
> # cryptsetup open /dev/nvme0n1p2 luks-test
> Enter passphrase for /dev/nvme0n1p2:
> [worked]
> # mount -o ro,rescue=usebackuproot  /dev/mapper/luks-test /mnt
> mount: /mnt: wrong fs type, bad option, bad superblock on 
> /dev/mapper/luks-test, missing codepage or helper program, or other error.
>         dmesg(1) may have more information after failed mount system call.
> 
>       dmesg after this mount attempt:
> [ 5179.422225] BTRFS info (device dm-1): flagging fs with big metadata 
> feature
> [ 5179.422248] BTRFS info (device dm-1): trying to use backup root at 
> mount time
> [ 5179.422255] BTRFS info (device dm-1): using free space tree
> [ 5179.422260] BTRFS info (device dm-1): has skinny extents
> [ 5179.431117] BTRFS error (device dm-1): parent transid verify failed 
> on 334692352 wanted 14761 found 14765
> [ 5179.431338] BTRFS error (device dm-1): parent transid verify failed 
> on 334692352 wanted 14761 found 14765
> [ 5179.431358] BTRFS error (device dm-1): failed to read block groups: -5
> [ 5179.433358] BTRFS error (device dm-1): open_ctree failed
> 
> # btrfs check /dev/mapper/luks-test 2>&1|less -I
> parent transid verify failed on 334692352 wanted 14761 found 14765
> parent transid verify failed on 334692352 wanted 14761 found 14765
> parent transid verify failed on 334692352 wanted 14761 found 14765
> Ignoring transid failure
> [1/7] checking root items
> parent transid verify failed on 334643200 wanted 14761 found 14765
> parent transid verify failed on 334643200 wanted 14761 found 14765
> parent transid verify failed on 334643200 wanted 14761 found 14765
> Ignoring transid failure
> parent transid verify failed on 334659584 wanted 14761 found 14765
> parent transid verify failed on 334659584 wanted 14761 found 14765
> parent transid verify failed on 334659584 wanted 14761 found 14765
> Ignoring transid failure
> parent transid verify failed on 334430208 wanted 6728 found 14763
> parent transid verify failed on 334430208 wanted 6728 found 14763
> parent transid verify failed on 334430208 wanted 6728 found 14763
> Ignoring transid failure
> parent transid verify failed on 334675968 wanted 14761 found 14765
> parent transid verify failed on 334675968 wanted 14761 found 14765
> parent transid verify failed on 334675968 wanted 14761 found 14765
> Ignoring transid failure
> parent transid verify failed on 335216640 wanted 6728 found 14765
> parent transid verify failed on 335216640 wanted 6728 found 14765
> parent transid verify failed on 335216640 wanted 6728 found 14765
> Ignoring transid failure
> parent transid verify failed on 320847872 wanted 14323 found 14763
> parent transid verify failed on 320847872 wanted 14323 found 14763
> parent transid verify failed on 320847872 wanted 14323 found 14763
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=119848960 item=49 parent 
> level=1 child bytenr=320847872 child level=1
> ERROR: failed to repair root items: Input/output error
> [2/7] checking extents
> parent transid verify failed on 340246528 wanted 14741 found 14764
> parent transid verify failed on 340246528 wanted 14741 found 14764
> parent transid verify failed on 340246528 wanted 14741 found 14764
> Ignoring transid failure
> [... skipping many lines]
> root 257 inode 1866942 errors 2001, no inode item, link count wrong
>          unresolved ref dir 5719 index 9016 namelen 28 name 
> SiteSecurityServiceState.txt filetype 1 errors 4, no inode ref
> root 257 inode 1866943 errors 2001, no inode item, link count wrong
>          unresolved ref dir 5719 index 9018 namelen 21 name 
> AlternateServices.txt filetype 1 errors 4, no inode ref
> root 257 inode 1866989 errors 2001, no inode item, link count wrong
>          unresolved ref dir 346 index 16043 namelen 4 name user filetype 
> 1 errors 4, no inode ref
> root 257 inode 1866990 errors 2001, no inode item, link count wrong
>          unresolved ref dir 1216 index 11701 namelen 46 name 
> 0_3_1920_1080_8b47947fd8179de11b12e22fa2a454c8 filetype 1 errors 4, no 
> inode ref
> root 257 inode 1866991 errors 2001, no inode item, link count wrong
>          unresolved ref dir 5720 index 6961 namelen 16 name 
> recovery.jsonlz4 filetype 1 errors 4, no inode ref
> root 257 inode 1866995 errors 2001, no inode item, link count wrong
>          unresolved ref dir 6765 index 134 namelen 42 name 
> 3647222921wleabcEoxlt-eengsairo.sqlite-wal filetype 1 errors 4, no inode 
> ref
> parent transid verify failed on 348258304 wanted 14749 found 14766
> Ignoring transid failure
> parent transid verify failed on 348258304 wanted 14749 found 14766
> Ignoring transid failure
> parent transid verify failed on 348258304 wanted 14749 found 14766
> Ignoring transid failure
> parent transid verify failed on 348258304 wanted 14749 found 14766
> Ignoring transid failure
> ERROR: errors found in fs roots
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/luks-test
> UUID: 2d1dc6b4-84ab-4c64-91a0-669b6228c516
> found 83720974336 bytes used, error(s) found
> total csum bytes: 50467580
> total tree bytes: 266665984
> total fs tree bytes: 186236928
> total extent tree bytes: 17317888
> btree space waste bytes: 40645922
> file data blocks allocated: 87345299456
>   referenced 47847440384
> [less: lines 1364572-1364611/1364611 byte 90015202/90015202 (END)  
> (press RETURN)]
> 
> # btrfs-find-root /dev/mapper/luks-test
> parent transid verify failed on 334692352 wanted 14761 found 14765
> parent transid verify failed on 334692352 wanted 14761 found 14765
> ERROR: failed to read block groups: Input/output error
> Superblock thinks the generation is 14761
> Superblock thinks the level is 0
> Found tree root at 444612608 gen 14761 level 0
> Well block 347160576(gen: 14768 level: 0) seems good, but 
> generation/level doesn't match, want gen: 14761 level: 0
> Well block 348192768(gen: 14766 level: 0) seems good, but 
> generation/level doesn't match, want gen: 14761 level: 0
> Well block 347865088(gen: 14765 level: 0) seems good, but 
> generation/level doesn't match, want gen: 14761 level: 0
> Well block 418840576(gen: 14758 level: 0) seems good, but 
> generation/level doesn't match, want gen: 14761 level: 0
> Well block 417120256(gen: 14749 level: 0) seems good, but 
> generation/level doesn't match, want gen: 14761 level: 0
> Well block 352256000(gen: 14743 level: 0) seems good, but 
> generation/level doesn't match, want gen: 14761 level: 0
> [end]
> 
> This is what I found out by reading.
> A fix - if possible - is out of my league now and don't want to poke 
> around and make things worse.
> Any chance to get this FS at least mounted for RO?

You can try "mount -o ro,rescue=all", which will skip the block group 
item search, but still there will be other corruptions.

I'm more interested in how this happened.
The main point here is, the found transid is newer than the on-disk 
transid, which means metadata COW is not working at all.

Are you using unsafe mount options like disabling barriers?

Thanks,
Qu

> 
> Thanks a lot, Michael.

Re: Corrupted btrfs (LUKS), seeking advice

[Qu Wenruo]

On 2022/8/9 01:33, Michael Zacherl wrote:
> On 8/8/22 16:57, Chris Murphy wrote:
>> mount -o ro,rescue=all
>>
>> If that works you can umount and try again adding all the rescue 
>> options except idatacsums.> It's nice to have datacsum warnings 
>> (unless there's just to many errors.)
> 
> rescue=all mounts.
> 
> dmesg from   mount -o 
> ro,rescue=usebackuproot,rescue=nologreplay,rescue=ignorebadroots 
> /dev/mapper/luks-test /mnt :
> 
> [20680.066631] BTRFS info (device dm-1): flagging fs with big metadata 
> feature
> [20680.066641] BTRFS info (device dm-1): trying to use backup root at 
> mount time
> [20680.066646] BTRFS info (device dm-1): disabling log replay at mount time
> [20680.066650] BTRFS info (device dm-1): ignoring bad roots
> [20680.066652] BTRFS info (device dm-1): using free space tree
> [20680.066654] BTRFS info (device dm-1): has skinny extents
> [20680.072359] BTRFS error (device dm-1): parent transid verify failed 
> on 334692352 wanted 14761 found 14765
> [20680.072571] BTRFS error (device dm-1): parent transid verify failed 
> on 334692352 wanted 14761 found 14765
> [20680.073296] BTRFS info (device dm-1): enabling ssd optimizations
> 
> A brief look shows I can access data!
> 
> When also omitting nologreplay the FS wouldn't mount and dmesg shows
> 
> [20837.557120] BTRFS info (device dm-1): flagging fs with big metadata 
> feature
> [20837.557138] BTRFS info (device dm-1): trying to use backup root at 
> mount time
> [20837.557148] BTRFS info (device dm-1): ignoring bad roots
> [20837.557152] BTRFS info (device dm-1): using free space tree
> [20837.557156] BTRFS info (device dm-1): has skinny extents
> [20837.567354] BTRFS error (device dm-1): parent transid verify failed 
> on 334692352 wanted 14761 found 14765
> [20837.567809] BTRFS error (device dm-1): parent transid verify failed 
> on 334692352 wanted 14761 found 14765
> [20837.569387] BTRFS info (device dm-1): enabling ssd optimizationsquite 
> understans
> [20837.569403] BTRFS info (device dm-1): start tree-log replay

Try this mount option "-o ro,rescue=all,rescue=nologreplay".

> [20837.637057] BTRFS error (device dm-1): parent transid verify failed 
> on 337412096 wanted 5492 found 14764
> [20837.637223] BTRFS error (device dm-1): parent transid verify failed 
> on 337412096 wanted 5492 found 14764
> [20837.656541] BTRFS error (device dm-1): parent transid verify failed 
> on 334643200 wanted 14761 found 14765
> [20837.656670] BTRFS error (device dm-1): parent transid verify failed 
> on 334643200 wanted 14761 found 14765
> [20837.656676] BTRFS: error (device dm-1: state A) in 
> btrfs_run_delayed_refs:2151: errno=-5 IO failure
> [20837.656682] BTRFS: error (device dm-1: state EA) in 
> btrfs_replay_log:2567: errno=-5 IO failure (Failed to recover log tree)
> [20837.675238] BTRFS error (device dm-1: state EA): open_ctree failed
> 
> Is this FS repairable to a usable state?

Definitely no.

Thanks,
Qu
> 
> Thanks a lot, Michael.
> 
> 
> 

Re: Corrupted btrfs (LUKS), seeking advice

[Michael Zacherl]

On 8/9/22 01:22, Qu Wenruo wrote:
> You can try "mount -o ro,rescue=all", which will skip the block group
> item search, but still there will be other corruptions.
>
> I'm more interested in how this happened.
> The main point here is, the found transid is newer than the on-disk
> transid, which means metadata COW is not working at all.
>
> Are you using unsafe mount options like disabling barriers?

No, this upgraded setup is fairly new and completely stock.
(and most of that terminology I have to look up anyway)
Btrfs is new to me and I don't experiment on systems I need for work.

I think what happened is having had mounted the FS twice by accident:
The former system (Mint 19.3/ext4) has been cloned to a USB-stick which I can boot from.
In one such session I mounted the new btrfs nvme on the old system for some data exchange.
I put the old system to hibernation but forgot to unmount the nvme prior to that. :(

So when booting up the new system from the nvme it was like having had a hard shutdown.
So that in itself wouldn't be the problem, I'd think.
But the other day I again booted from the old system from its hibernation with the forgotten nvme mounted.
And that was the killer, I'd say, since a lot of metadata has changed on that btrfs meanwhile.

On top of it, btrfs is v4.15.1 on the old system, many things just don't exits on this version, AFAICT.
If that made things worse, I can't say.

On 8/9/22 01:24, Qu Wenruo wrote:
> Try this mount option "-o ro,rescue=all,rescue=nologreplay".

Oh, I thought "nologgreplay" would be included in "all"?

>> Is this FS repairable to a usable state?
> 
> Definitely no.

Ouch. But meanwhile I can see the damage I did to it.
I'm currently abroad, so no access to my regular backup infrastructure.

However, since I've to re-install the new system when I'm back:
There would be enough space on the new ssd for a second partition, which I'd like not to go for.
Or is there an option for additional redundancy within the same btrfs to help in case of such a bad mistake (I'd dearly try to avoid anyway, but ...)?

Thanks a lot for your time, Michael.

Re: Corrupted btrfs (LUKS), seeking advice

[Qu Wenruo]

On 2022/8/9 19:23, Michael Zacherl wrote:
> On 8/9/22 01:22, Qu Wenruo wrote:
>> You can try "mount -o ro,rescue=all", which will skip the block group
>> item search, but still there will be other corruptions.
>>
>> I'm more interested in how this happened.
>> The main point here is, the found transid is newer than the on-disk
>> transid, which means metadata COW is not working at all.
>>
>> Are you using unsafe mount options like disabling barriers?
>
> No, this upgraded setup is fairly new and completely stock.
> (and most of that terminology I have to look up anyway)
> Btrfs is new to me and I don't experiment on systems I need for work.
>
> I think what happened is having had mounted the FS twice by accident:
> The former system (Mint 19.3/ext4) has been cloned to a USB-stick which
> I can boot from.
> In one such session I mounted the new btrfs nvme on the old system for
> some data exchange.
> I put the old system to hibernation but forgot to unmount the nvme prior
> to that. :(

Hibernation, I'm not sure about the details, but it looks like there
were some corruption reports related to that.

>
> So when booting up the new system from the nvme it was like having had a
> hard shutdown.

A hard shutdown to btrfs itself should not cause anything wrong, that's
ensured by its mandatory metadata COW.

> So that in itself wouldn't be the problem, I'd think.
> But the other day I again booted from the old system from its
> hibernation with the forgotten nvme mounted.

Oh I got it now, it's not after the hibernation immediately, but the
resume from hibernation, and then some write into the already
out-of-sync fs caused the problem.

> And that was the killer, I'd say, since a lot of metadata has changed on
> that btrfs meanwhile.

Yes, I believe that's the case.

>
> On top of it, btrfs is v4.15.1 on the old system, many things just don't
> exits on this version, AFAICT.
> If that made things worse, I can't say.
>
> On 8/9/22 01:24, Qu Wenruo wrote:
>> Try this mount option "-o ro,rescue=all,rescue=nologreplay".
>
> Oh, I thought "nologgreplay" would be included in "all"?

I checked the code, and the latest code is indeed including that.

But that's weird, if included we should not try to replay the log, thus
that "start tree-log replay" should not occur.

Anyway if rescue=all doesn't work, you may try "btrfs rescue zero-log"
to manually delete that dirty log and then RO mount should still be
possible.

>
>>> Is this FS repairable to a usable state?
>>
>> Definitely no.
>
> Ouch. But meanwhile I can see the damage I did to it.
> I'm currently abroad, so no access to my regular backup infrastructure.
>
> However, since I've to re-install the new system when I'm back:
> There would be enough space on the new ssd for a second partition, which
> I'd like not to go for.
> Or is there an option for additional redundancy within the same btrfs to
> help in case of such a bad mistake (I'd dearly try to avoid anyway, but
> ...)?

I'm not sure if there is anyway to prevent such out-of-sync RW mount
from corrupting the fs.

We can go RAID1 (if you have multiple disks) or DUP (single device,
double copy), but none of them can handle the case you hit.

Personally speaking, I never trust hibernation/suspension, although due
to other ACPI related reasons.
Now I won't even touch hibernation/suspension at all, just to avoid any
removable storage corruption.

Thanks,
Qu

>
> Thanks a lot for your time, Michael.
>
>

Re: Corrupted btrfs (LUKS), seeking advice

[Andrei Borzenkov]

On 09.08.2022 14:37, Qu Wenruo wrote:
...
>>
>> I think what happened is having had mounted the FS twice by accident:
>> The former system (Mint 19.3/ext4) has been cloned to a USB-stick which
>> I can boot from.
>> In one such session I mounted the new btrfs nvme on the old system for
>> some data exchange.
>> I put the old system to hibernation but forgot to unmount the nvme prior
>> to that. :(
> 
> Hibernation, I'm not sure about the details, but it looks like there
> were some corruption reports related to that.
> 
>>
>> So when booting up the new system from the nvme it was like having had a
>> hard shutdown.
> 
> A hard shutdown to btrfs itself should not cause anything wrong, that's
> ensured by its mandatory metadata COW.
> 
>> So that in itself wouldn't be the problem, I'd think.
>> But the other day I again booted from the old system from its
>> hibernation with the forgotten nvme mounted.
> 
> Oh I got it now, it's not after the hibernation immediately, but the
> resume from hibernation, and then some write into the already
> out-of-sync fs caused the problem.
> 
>> And that was the killer, I'd say, since a lot of metadata has changed on
>> that btrfs meanwhile.
> 
> Yes, I believe that's the case.
> 
...
> 
> Personally speaking, I never trust hibernation/suspension, although due
> to other ACPI related reasons.
> Now I won't even touch hibernation/suspension at all, just to avoid any
> removable storage corruption.
> 

I wonder if it is possible to add resume hook to check that on-disk
state matches in-memory state and go read-only if on-disk state changed.
Checking current generation should be enough to detect it?

Re: Corrupted btrfs (LUKS), seeking advice

[Qu Wenruo]

On 2022/8/10 14:10, Andrei Borzenkov wrote:
> On 09.08.2022 14:37, Qu Wenruo wrote:
> ...
>>>
>>> I think what happened is having had mounted the FS twice by accident:
>>> The former system (Mint 19.3/ext4) has been cloned to a USB-stick which
>>> I can boot from.
>>> In one such session I mounted the new btrfs nvme on the old system for
>>> some data exchange.
>>> I put the old system to hibernation but forgot to unmount the nvme prior
>>> to that. :(
>>
>> Hibernation, I'm not sure about the details, but it looks like there
>> were some corruption reports related to that.
>>
>>>
>>> So when booting up the new system from the nvme it was like having had a
>>> hard shutdown.
>>
>> A hard shutdown to btrfs itself should not cause anything wrong, that's
>> ensured by its mandatory metadata COW.
>>
>>> So that in itself wouldn't be the problem, I'd think.
>>> But the other day I again booted from the old system from its
>>> hibernation with the forgotten nvme mounted.
>>
>> Oh I got it now, it's not after the hibernation immediately, but the
>> resume from hibernation, and then some write into the already
>> out-of-sync fs caused the problem.
>>
>>> And that was the killer, I'd say, since a lot of metadata has changed on
>>> that btrfs meanwhile.
>>
>> Yes, I believe that's the case.
>>
> ...
>>
>> Personally speaking, I never trust hibernation/suspension, although due
>> to other ACPI related reasons.
>> Now I won't even touch hibernation/suspension at all, just to avoid any
>> removable storage corruption.
>>
>
> I wonder if it is possible to add resume hook to check that on-disk
> state matches in-memory state and go read-only if on-disk state changed.

AFAIK what we should do is, using hooks to unmount all removable disks,
before entering suspension/hibernation.

> Checking current generation should be enough to detect it?

IMHO fs doesn't really have any way to know we're going into
hibernation/suspension.

If we have such mechanism, then yes it's possible.

Thanks,
Qu

Re: Corrupted btrfs (LUKS), seeking advice

[Goffredo Baroncelli]

On 10/08/2022 08.29, Qu Wenruo wrote:
>
>
> On 2022/8/10 14:10, Andrei Borzenkov wrote:
>> On 09.08.2022 14:37, Qu Wenruo wrote:
>> ...
>>>>
>>>> I think what happened is having had mounted the FS twice by accident:
>>>> The former system (Mint 19.3/ext4) has been cloned to a USB-stick which
>>>> I can boot from.
>>>> In one such session I mounted the new btrfs nvme on the old system for
>>>> some data exchange.
>>>> I put the old system to hibernation but forgot to unmount the nvme prior
>>>> to that. :(
>>>
>>> Hibernation, I'm not sure about the details, but it looks like there
>>> were some corruption reports related to that.
>>>
>>>>
>>>> So when booting up the new system from the nvme it was like having had a
>>>> hard shutdown.
>>>
>>> A hard shutdown to btrfs itself should not cause anything wrong, that's
>>> ensured by its mandatory metadata COW.
>>>
>>>> So that in itself wouldn't be the problem, I'd think.
>>>> But the other day I again booted from the old system from its
>>>> hibernation with the forgotten nvme mounted.
>>>
>>> Oh I got it now, it's not after the hibernation immediately, but the
>>> resume from hibernation, and then some write into the already
>>> out-of-sync fs caused the problem.
>>>
>>>> And that was the killer, I'd say, since a lot of metadata has changed on
>>>> that btrfs meanwhile.
>>>
>>> Yes, I believe that's the case.
>>>
>> ...
>>>
>>> Personally speaking, I never trust hibernation/suspension, although due
>>> to other ACPI related reasons.
>>> Now I won't even touch hibernation/suspension at all, just to avoid any
>>> removable storage corruption.
>>>
>>
>> I wonder if it is possible to add resume hook to check that on-disk
>> state matches in-memory state and go read-only if on-disk state changed.
>
> AFAIK what we should do is, using hooks to unmount all removable disks,
> before entering suspension/hibernation.

I don't think that this is doable, because before the hibernation there are

some file descriptors opened, so a full unmount is not an option.


>
>> Checking current generation should be enough to detect it?
>
> IMHO fs doesn't really have any way to know we're going into
> hibernation/suspension.
>
> If we have such mechanism, then yes it's possible.

I think that the only available hooks are

	struct super_operations.freeze_fs 	struct super_operations.unfreeze_fs

I think a check about a consistency of the transaction id should be appropriate, and then if it fails do a panic.
But the case of Michael seems a "misuse" to care of: check that all the disks have the same transaction ids.


>
> Thanks,
> Qu


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: Corrupted btrfs (LUKS), seeking advice

[Qu Wenruo]

On 2022/8/10 15:26, Goffredo Baroncelli wrote:
> On 10/08/2022 08.29, Qu Wenruo wrote:
>>
>>
>> On 2022/8/10 14:10, Andrei Borzenkov wrote:
>>> On 09.08.2022 14:37, Qu Wenruo wrote:
>>> ...
>>>>>
>>>>> I think what happened is having had mounted the FS twice by accident:
>>>>> The former system (Mint 19.3/ext4) has been cloned to a USB-stick
>>>>> which
>>>>> I can boot from.
>>>>> In one such session I mounted the new btrfs nvme on the old system for
>>>>> some data exchange.
>>>>> I put the old system to hibernation but forgot to unmount the nvme
>>>>> prior
>>>>> to that. :(
>>>>
>>>> Hibernation, I'm not sure about the details, but it looks like there
>>>> were some corruption reports related to that.
>>>>
>>>>>
>>>>> So when booting up the new system from the nvme it was like having
>>>>> had a
>>>>> hard shutdown.
>>>>
>>>> A hard shutdown to btrfs itself should not cause anything wrong, that's
>>>> ensured by its mandatory metadata COW.
>>>>
>>>>> So that in itself wouldn't be the problem, I'd think.
>>>>> But the other day I again booted from the old system from its
>>>>> hibernation with the forgotten nvme mounted.
>>>>
>>>> Oh I got it now, it's not after the hibernation immediately, but the
>>>> resume from hibernation, and then some write into the already
>>>> out-of-sync fs caused the problem.
>>>>
>>>>> And that was the killer, I'd say, since a lot of metadata has
>>>>> changed on
>>>>> that btrfs meanwhile.
>>>>
>>>> Yes, I believe that's the case.
>>>>
>>> ...
>>>>
>>>> Personally speaking, I never trust hibernation/suspension, although due
>>>> to other ACPI related reasons.
>>>> Now I won't even touch hibernation/suspension at all, just to avoid any
>>>> removable storage corruption.
>>>>
>>>
>>> I wonder if it is possible to add resume hook to check that on-disk
>>> state matches in-memory state and go read-only if on-disk state changed.
>>
>> AFAIK what we should do is, using hooks to unmount all removable disks,
>> before entering suspension/hibernation.
>
> I don't think that this is doable, because before the hibernation there are
>
> some file descriptors opened, so a full unmount is not an option.
>
>
>>
>>> Checking current generation should be enough to detect it?
>>
>> IMHO fs doesn't really have any way to know we're going into
>> hibernation/suspension.
>>
>> If we have such mechanism, then yes it's possible.
>
> I think that the only available hooks are
>
>      struct super_operations.freeze_fs     struct
> super_operations.unfreeze_fs
>
> I think a check about a consistency of the transaction id should be
> appropriate, and then if it fails do a panic.
> But the case of Michael seems a "misuse" to care of: check that all the
> disks have the same transaction ids.

I think your idea is pretty to the point.

If we found some obvious unexpected modification, we can mark the FS RO
to minimize the damage (as long as the unexpected modification is still
correct, the fs will be safe).

I just sent a patch based on your idea, feel free to add your
Suggested-by: and other tags:
https://lore.kernel.org/linux-btrfs/41c43c7d6aa25af13391905061e2d203f7853382.1660199812.git.wqu@suse.com/T/#u

Thanks,
Qu

>
>
>>
>> Thanks,
>> Qu
>
>

Re: Corrupted btrfs (LUKS), seeking advice

[Michael Zacherl]

On 11.08.2022, at 08:40, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>> If we have such mechanism, then yes it's possible.
>> 
>> I think that the only available hooks are
>> 
>>     struct super_operations.freeze_fs     struct
>> super_operations.unfreeze_fs
>> 
>> I think a check about a consistency of the transaction id should be
>> appropriate, and then if it fails do a panic.
>> But the case of Michael seems a "misuse" to care of: check that all the
>> disks have the same transaction ids.

“misuse” indeed, I have a tendency (predisposition?) to exhaust technical possibilities. 🙄

> I think your idea is pretty to the point.
> 
> If we found some obvious unexpected modification, we can mark the FS RO
> to minimize the damage (as long as the unexpected modification is still
> correct, the fs will be safe).

I observed this (user POV) behaviour on an old ext4 system:
After waking the hibernating ext4 system again the FS was mounted RO and the OS unusable.
Rebooting dropped into initramfs prompt and an fsck was needed.
Then booting could be continued.

> I just sent a patch based on your idea, feel free to add your
> Suggested-by: and other tags:
> https://lore.kernel.org/linux-btrfs/41c43c7d6aa25af13391905061e2d203f7853382.1660199812.git.wqu@suse.com/T/#u

Thank you very much to you and all for all your efforts - specifically and in general! 
Michael.