* Corrupted btrfs (LUKS), seeking advice
@ 2022-08-08 13:06 Michael Zacherl
2022-08-08 16:57 ` Chris Murphy
2022-08-09 0:22 ` Qu Wenruo
0 siblings, 2 replies; 12+ messages in thread
From: Michael Zacherl @ 2022-08-08 13:06 UTC (permalink / raw)
To: linux-btrfs
Hello,
on the occasion of retrofitting a 2TB ssd for my old XPS13 9350 I decided to give btrfs w/ encryption a try (this was in June).
Now, by a dumb mistake, I have a corrupted btrfs (LUKS encrypted).
Since I can't boot from this partition anymore I'm using the distro's live system.
This partition can't be mounted.
# uname -a
Linux EndeavourOS 5.18.5-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 16 Jun 2022 20:40:45 +0000 x86_64 GNU/Linux
# btrfs --version
btrfs-progs v5.18.1
What I did so far:
# cryptsetup open /dev/nvme0n1p2 luks-test
Enter passphrase for /dev/nvme0n1p2:
[worked]
# mount -o ro,rescue=usebackuproot /dev/mapper/luks-test /mnt
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/mapper/luks-test, missing codepage or helper program, or other error.
dmesg(1) may have more information after failed mount system call.
dmesg after this mount attempt:
[ 5179.422225] BTRFS info (device dm-1): flagging fs with big metadata feature
[ 5179.422248] BTRFS info (device dm-1): trying to use backup root at mount time
[ 5179.422255] BTRFS info (device dm-1): using free space tree
[ 5179.422260] BTRFS info (device dm-1): has skinny extents
[ 5179.431117] BTRFS error (device dm-1): parent transid verify failed on 334692352 wanted 14761 found 14765
[ 5179.431338] BTRFS error (device dm-1): parent transid verify failed on 334692352 wanted 14761 found 14765
[ 5179.431358] BTRFS error (device dm-1): failed to read block groups: -5
[ 5179.433358] BTRFS error (device dm-1): open_ctree failed
# btrfs check /dev/mapper/luks-test 2>&1|less -I
parent transid verify failed on 334692352 wanted 14761 found 14765
parent transid verify failed on 334692352 wanted 14761 found 14765
parent transid verify failed on 334692352 wanted 14761 found 14765
Ignoring transid failure
[1/7] checking root items
parent transid verify failed on 334643200 wanted 14761 found 14765
parent transid verify failed on 334643200 wanted 14761 found 14765
parent transid verify failed on 334643200 wanted 14761 found 14765
Ignoring transid failure
parent transid verify failed on 334659584 wanted 14761 found 14765
parent transid verify failed on 334659584 wanted 14761 found 14765
parent transid verify failed on 334659584 wanted 14761 found 14765
Ignoring transid failure
parent transid verify failed on 334430208 wanted 6728 found 14763
parent transid verify failed on 334430208 wanted 6728 found 14763
parent transid verify failed on 334430208 wanted 6728 found 14763
Ignoring transid failure
parent transid verify failed on 334675968 wanted 14761 found 14765
parent transid verify failed on 334675968 wanted 14761 found 14765
parent transid verify failed on 334675968 wanted 14761 found 14765
Ignoring transid failure
parent transid verify failed on 335216640 wanted 6728 found 14765
parent transid verify failed on 335216640 wanted 6728 found 14765
parent transid verify failed on 335216640 wanted 6728 found 14765
Ignoring transid failure
parent transid verify failed on 320847872 wanted 14323 found 14763
parent transid verify failed on 320847872 wanted 14323 found 14763
parent transid verify failed on 320847872 wanted 14323 found 14763
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=119848960 item=49 parent level=1 child bytenr=320847872 child level=1
ERROR: failed to repair root items: Input/output error
[2/7] checking extents
parent transid verify failed on 340246528 wanted 14741 found 14764
parent transid verify failed on 340246528 wanted 14741 found 14764
parent transid verify failed on 340246528 wanted 14741 found 14764
Ignoring transid failure
[... skipping many lines]
root 257 inode 1866942 errors 2001, no inode item, link count wrong
unresolved ref dir 5719 index 9016 namelen 28 name SiteSecurityServiceState.txt filetype 1 errors 4, no inode ref
root 257 inode 1866943 errors 2001, no inode item, link count wrong
unresolved ref dir 5719 index 9018 namelen 21 name AlternateServices.txt filetype 1 errors 4, no inode ref
root 257 inode 1866989 errors 2001, no inode item, link count wrong
unresolved ref dir 346 index 16043 namelen 4 name user filetype 1 errors 4, no inode ref
root 257 inode 1866990 errors 2001, no inode item, link count wrong
unresolved ref dir 1216 index 11701 namelen 46 name 0_3_1920_1080_8b47947fd8179de11b12e22fa2a454c8 filetype 1 errors 4, no inode ref
root 257 inode 1866991 errors 2001, no inode item, link count wrong
unresolved ref dir 5720 index 6961 namelen 16 name recovery.jsonlz4 filetype 1 errors 4, no inode ref
root 257 inode 1866995 errors 2001, no inode item, link count wrong
unresolved ref dir 6765 index 134 namelen 42 name 3647222921wleabcEoxlt-eengsairo.sqlite-wal filetype 1 errors 4, no inode ref
parent transid verify failed on 348258304 wanted 14749 found 14766
Ignoring transid failure
parent transid verify failed on 348258304 wanted 14749 found 14766
Ignoring transid failure
parent transid verify failed on 348258304 wanted 14749 found 14766
Ignoring transid failure
parent transid verify failed on 348258304 wanted 14749 found 14766
Ignoring transid failure
ERROR: errors found in fs roots
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-test
UUID: 2d1dc6b4-84ab-4c64-91a0-669b6228c516
found 83720974336 bytes used, error(s) found
total csum bytes: 50467580
total tree bytes: 266665984
total fs tree bytes: 186236928
total extent tree bytes: 17317888
btree space waste bytes: 40645922
file data blocks allocated: 87345299456
referenced 47847440384
[less: lines 1364572-1364611/1364611 byte 90015202/90015202 (END) (press RETURN)]
# btrfs-find-root /dev/mapper/luks-test
parent transid verify failed on 334692352 wanted 14761 found 14765
parent transid verify failed on 334692352 wanted 14761 found 14765
ERROR: failed to read block groups: Input/output error
Superblock thinks the generation is 14761
Superblock thinks the level is 0
Found tree root at 444612608 gen 14761 level 0
Well block 347160576(gen: 14768 level: 0) seems good, but generation/level doesn't match, want gen: 14761 level: 0
Well block 348192768(gen: 14766 level: 0) seems good, but generation/level doesn't match, want gen: 14761 level: 0
Well block 347865088(gen: 14765 level: 0) seems good, but generation/level doesn't match, want gen: 14761 level: 0
Well block 418840576(gen: 14758 level: 0) seems good, but generation/level doesn't match, want gen: 14761 level: 0
Well block 417120256(gen: 14749 level: 0) seems good, but generation/level doesn't match, want gen: 14761 level: 0
Well block 352256000(gen: 14743 level: 0) seems good, but generation/level doesn't match, want gen: 14761 level: 0
[end]
This is what I found out by reading.
A fix - if possible - is out of my league now and don't want to poke around and make things worse.
Any chance to get this FS at least mounted for RO?
Thanks a lot, Michael.
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: Corrupted btrfs (LUKS), seeking advice
2022-08-08 13:06 Corrupted btrfs (LUKS), seeking advice Michael Zacherl
@ 2022-08-08 16:57 ` Chris Murphy
2022-08-08 17:33 ` Michael Zacherl
2022-08-09 0:22 ` Qu Wenruo
1 sibling, 1 reply; 12+ messages in thread
From: Chris Murphy @ 2022-08-08 16:57 UTC (permalink / raw)
To: Michael Zacherl, Btrfs BTRFS
On Mon, Aug 8, 2022, at 9:06 AM, Michael Zacherl wrote:
>
> # mount -o ro,rescue=usebackuproot /dev/mapper/luks-test /mnt
Try
mount -o ro,rescue=all
If that works you can umount and try again adding all the rescue options except idatacsums. It's nice to have datacsum warnings (unless there's just to many errors.)
Otherwise its btrfs restore, which is tedious to use but you should still be able to get data off.
--
Chris Murphy
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Corrupted btrfs (LUKS), seeking advice
2022-08-08 16:57 ` Chris Murphy
@ 2022-08-08 17:33 ` Michael Zacherl
2022-08-09 0:24 ` Qu Wenruo
0 siblings, 1 reply; 12+ messages in thread
From: Michael Zacherl @ 2022-08-08 17:33 UTC (permalink / raw)
To: Btrfs BTRFS
On 8/8/22 16:57, Chris Murphy wrote:
> mount -o ro,rescue=all
>
> If that works you can umount and try again adding all the rescue options except idatacsums.> It's nice to have datacsum warnings (unless there's just to many errors.)
rescue=all mounts.
dmesg from mount -o ro,rescue=usebackuproot,rescue=nologreplay,rescue=ignorebadroots /dev/mapper/luks-test /mnt :
[20680.066631] BTRFS info (device dm-1): flagging fs with big metadata feature
[20680.066641] BTRFS info (device dm-1): trying to use backup root at mount time
[20680.066646] BTRFS info (device dm-1): disabling log replay at mount time
[20680.066650] BTRFS info (device dm-1): ignoring bad roots
[20680.066652] BTRFS info (device dm-1): using free space tree
[20680.066654] BTRFS info (device dm-1): has skinny extents
[20680.072359] BTRFS error (device dm-1): parent transid verify failed on 334692352 wanted 14761 found 14765
[20680.072571] BTRFS error (device dm-1): parent transid verify failed on 334692352 wanted 14761 found 14765
[20680.073296] BTRFS info (device dm-1): enabling ssd optimizations
A brief look shows I can access data!
When also omitting nologreplay the FS wouldn't mount and dmesg shows
[20837.557120] BTRFS info (device dm-1): flagging fs with big metadata feature
[20837.557138] BTRFS info (device dm-1): trying to use backup root at mount time
[20837.557148] BTRFS info (device dm-1): ignoring bad roots
[20837.557152] BTRFS info (device dm-1): using free space tree
[20837.557156] BTRFS info (device dm-1): has skinny extents
[20837.567354] BTRFS error (device dm-1): parent transid verify failed on 334692352 wanted 14761 found 14765
[20837.567809] BTRFS error (device dm-1): parent transid verify failed on 334692352 wanted 14761 found 14765
[20837.569387] BTRFS info (device dm-1): enabling ssd optimizationsquite understans
[20837.569403] BTRFS info (device dm-1): start tree-log replay
[20837.637057] BTRFS error (device dm-1): parent transid verify failed on 337412096 wanted 5492 found 14764
[20837.637223] BTRFS error (device dm-1): parent transid verify failed on 337412096 wanted 5492 found 14764
[20837.656541] BTRFS error (device dm-1): parent transid verify failed on 334643200 wanted 14761 found 14765
[20837.656670] BTRFS error (device dm-1): parent transid verify failed on 334643200 wanted 14761 found 14765
[20837.656676] BTRFS: error (device dm-1: state A) in btrfs_run_delayed_refs:2151: errno=-5 IO failure
[20837.656682] BTRFS: error (device dm-1: state EA) in btrfs_replay_log:2567: errno=-5 IO failure (Failed to recover log tree)
[20837.675238] BTRFS error (device dm-1: state EA): open_ctree failed
Is this FS repairable to a usable state?
Thanks a lot, Michael.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Corrupted btrfs (LUKS), seeking advice
2022-08-08 17:33 ` Michael Zacherl
@ 2022-08-09 0:24 ` Qu Wenruo
2022-08-09 11:23 ` Michael Zacherl
0 siblings, 1 reply; 12+ messages in thread
From: Qu Wenruo @ 2022-08-09 0:24 UTC (permalink / raw)
To: Michael Zacherl, Btrfs BTRFS
On 2022/8/9 01:33, Michael Zacherl wrote:
> On 8/8/22 16:57, Chris Murphy wrote:
>> mount -o ro,rescue=all
>>
>> If that works you can umount and try again adding all the rescue
>> options except idatacsums.> It's nice to have datacsum warnings
>> (unless there's just to many errors.)
>
> rescue=all mounts.
>
> dmesg from mount -o
> ro,rescue=usebackuproot,rescue=nologreplay,rescue=ignorebadroots
> /dev/mapper/luks-test /mnt :
>
> [20680.066631] BTRFS info (device dm-1): flagging fs with big metadata
> feature
> [20680.066641] BTRFS info (device dm-1): trying to use backup root at
> mount time
> [20680.066646] BTRFS info (device dm-1): disabling log replay at mount time
> [20680.066650] BTRFS info (device dm-1): ignoring bad roots
> [20680.066652] BTRFS info (device dm-1): using free space tree
> [20680.066654] BTRFS info (device dm-1): has skinny extents
> [20680.072359] BTRFS error (device dm-1): parent transid verify failed
> on 334692352 wanted 14761 found 14765
> [20680.072571] BTRFS error (device dm-1): parent transid verify failed
> on 334692352 wanted 14761 found 14765
> [20680.073296] BTRFS info (device dm-1): enabling ssd optimizations
>
> A brief look shows I can access data!
>
> When also omitting nologreplay the FS wouldn't mount and dmesg shows
>
> [20837.557120] BTRFS info (device dm-1): flagging fs with big metadata
> feature
> [20837.557138] BTRFS info (device dm-1): trying to use backup root at
> mount time
> [20837.557148] BTRFS info (device dm-1): ignoring bad roots
> [20837.557152] BTRFS info (device dm-1): using free space tree
> [20837.557156] BTRFS info (device dm-1): has skinny extents
> [20837.567354] BTRFS error (device dm-1): parent transid verify failed
> on 334692352 wanted 14761 found 14765
> [20837.567809] BTRFS error (device dm-1): parent transid verify failed
> on 334692352 wanted 14761 found 14765
> [20837.569387] BTRFS info (device dm-1): enabling ssd optimizationsquite
> understans
> [20837.569403] BTRFS info (device dm-1): start tree-log replay
Try this mount option "-o ro,rescue=all,rescue=nologreplay".
> [20837.637057] BTRFS error (device dm-1): parent transid verify failed
> on 337412096 wanted 5492 found 14764
> [20837.637223] BTRFS error (device dm-1): parent transid verify failed
> on 337412096 wanted 5492 found 14764
> [20837.656541] BTRFS error (device dm-1): parent transid verify failed
> on 334643200 wanted 14761 found 14765
> [20837.656670] BTRFS error (device dm-1): parent transid verify failed
> on 334643200 wanted 14761 found 14765
> [20837.656676] BTRFS: error (device dm-1: state A) in
> btrfs_run_delayed_refs:2151: errno=-5 IO failure
> [20837.656682] BTRFS: error (device dm-1: state EA) in
> btrfs_replay_log:2567: errno=-5 IO failure (Failed to recover log tree)
> [20837.675238] BTRFS error (device dm-1: state EA): open_ctree failed
>
> Is this FS repairable to a usable state?
Definitely no.
Thanks,
Qu
>
> Thanks a lot, Michael.
>
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Corrupted btrfs (LUKS), seeking advice
2022-08-09 0:24 ` Qu Wenruo
@ 2022-08-09 11:23 ` Michael Zacherl
2022-08-09 11:37 ` Qu Wenruo
0 siblings, 1 reply; 12+ messages in thread
From: Michael Zacherl @ 2022-08-09 11:23 UTC (permalink / raw)
To: linux-btrfs; +Cc: wqu
On 8/9/22 01:22, Qu Wenruo wrote:
> You can try "mount -o ro,rescue=all", which will skip the block group
> item search, but still there will be other corruptions.
>
> I'm more interested in how this happened.
> The main point here is, the found transid is newer than the on-disk
> transid, which means metadata COW is not working at all.
>
> Are you using unsafe mount options like disabling barriers?
No, this upgraded setup is fairly new and completely stock.
(and most of that terminology I have to look up anyway)
Btrfs is new to me and I don't experiment on systems I need for work.
I think what happened is having had mounted the FS twice by accident:
The former system (Mint 19.3/ext4) has been cloned to a USB-stick which I can boot from.
In one such session I mounted the new btrfs nvme on the old system for some data exchange.
I put the old system to hibernation but forgot to unmount the nvme prior to that. :(
So when booting up the new system from the nvme it was like having had a hard shutdown.
So that in itself wouldn't be the problem, I'd think.
But the other day I again booted from the old system from its hibernation with the forgotten nvme mounted.
And that was the killer, I'd say, since a lot of metadata has changed on that btrfs meanwhile.
On top of it, btrfs is v4.15.1 on the old system, many things just don't exits on this version, AFAICT.
If that made things worse, I can't say.
On 8/9/22 01:24, Qu Wenruo wrote:
> Try this mount option "-o ro,rescue=all,rescue=nologreplay".
Oh, I thought "nologgreplay" would be included in "all"?
>> Is this FS repairable to a usable state?
>
> Definitely no.
Ouch. But meanwhile I can see the damage I did to it.
I'm currently abroad, so no access to my regular backup infrastructure.
However, since I've to re-install the new system when I'm back:
There would be enough space on the new ssd for a second partition, which I'd like not to go for.
Or is there an option for additional redundancy within the same btrfs to help in case of such a bad mistake (I'd dearly try to avoid anyway, but ...)?
Thanks a lot for your time, Michael.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Corrupted btrfs (LUKS), seeking advice
2022-08-09 11:23 ` Michael Zacherl
@ 2022-08-09 11:37 ` Qu Wenruo
2022-08-10 6:10 ` Andrei Borzenkov
0 siblings, 1 reply; 12+ messages in thread
From: Qu Wenruo @ 2022-08-09 11:37 UTC (permalink / raw)
To: linux-btrfs; +Cc: wqu
On 2022/8/9 19:23, Michael Zacherl wrote:
> On 8/9/22 01:22, Qu Wenruo wrote:
>> You can try "mount -o ro,rescue=all", which will skip the block group
>> item search, but still there will be other corruptions.
>>
>> I'm more interested in how this happened.
>> The main point here is, the found transid is newer than the on-disk
>> transid, which means metadata COW is not working at all.
>>
>> Are you using unsafe mount options like disabling barriers?
>
> No, this upgraded setup is fairly new and completely stock.
> (and most of that terminology I have to look up anyway)
> Btrfs is new to me and I don't experiment on systems I need for work.
>
> I think what happened is having had mounted the FS twice by accident:
> The former system (Mint 19.3/ext4) has been cloned to a USB-stick which
> I can boot from.
> In one such session I mounted the new btrfs nvme on the old system for
> some data exchange.
> I put the old system to hibernation but forgot to unmount the nvme prior
> to that. :(
Hibernation, I'm not sure about the details, but it looks like there
were some corruption reports related to that.
>
> So when booting up the new system from the nvme it was like having had a
> hard shutdown.
A hard shutdown to btrfs itself should not cause anything wrong, that's
ensured by its mandatory metadata COW.
> So that in itself wouldn't be the problem, I'd think.
> But the other day I again booted from the old system from its
> hibernation with the forgotten nvme mounted.
Oh I got it now, it's not after the hibernation immediately, but the
resume from hibernation, and then some write into the already
out-of-sync fs caused the problem.
> And that was the killer, I'd say, since a lot of metadata has changed on
> that btrfs meanwhile.
Yes, I believe that's the case.
>
> On top of it, btrfs is v4.15.1 on the old system, many things just don't
> exits on this version, AFAICT.
> If that made things worse, I can't say.
>
> On 8/9/22 01:24, Qu Wenruo wrote:
>> Try this mount option "-o ro,rescue=all,rescue=nologreplay".
>
> Oh, I thought "nologgreplay" would be included in "all"?
I checked the code, and the latest code is indeed including that.
But that's weird, if included we should not try to replay the log, thus
that "start tree-log replay" should not occur.
Anyway if rescue=all doesn't work, you may try "btrfs rescue zero-log"
to manually delete that dirty log and then RO mount should still be
possible.
>
>>> Is this FS repairable to a usable state?
>>
>> Definitely no.
>
> Ouch. But meanwhile I can see the damage I did to it.
> I'm currently abroad, so no access to my regular backup infrastructure.
>
> However, since I've to re-install the new system when I'm back:
> There would be enough space on the new ssd for a second partition, which
> I'd like not to go for.
> Or is there an option for additional redundancy within the same btrfs to
> help in case of such a bad mistake (I'd dearly try to avoid anyway, but
> ...)?
I'm not sure if there is anyway to prevent such out-of-sync RW mount
from corrupting the fs.
We can go RAID1 (if you have multiple disks) or DUP (single device,
double copy), but none of them can handle the case you hit.
Personally speaking, I never trust hibernation/suspension, although due
to other ACPI related reasons.
Now I won't even touch hibernation/suspension at all, just to avoid any
removable storage corruption.
Thanks,
Qu
>
> Thanks a lot for your time, Michael.
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Corrupted btrfs (LUKS), seeking advice
2022-08-09 11:37 ` Qu Wenruo
@ 2022-08-10 6:10 ` Andrei Borzenkov
2022-08-10 6:29 ` Qu Wenruo
0 siblings, 1 reply; 12+ messages in thread
From: Andrei Borzenkov @ 2022-08-10 6:10 UTC (permalink / raw)
To: Qu Wenruo, linux-btrfs; +Cc: wqu
On 09.08.2022 14:37, Qu Wenruo wrote:
...
>>
>> I think what happened is having had mounted the FS twice by accident:
>> The former system (Mint 19.3/ext4) has been cloned to a USB-stick which
>> I can boot from.
>> In one such session I mounted the new btrfs nvme on the old system for
>> some data exchange.
>> I put the old system to hibernation but forgot to unmount the nvme prior
>> to that. :(
>
> Hibernation, I'm not sure about the details, but it looks like there
> were some corruption reports related to that.
>
>>
>> So when booting up the new system from the nvme it was like having had a
>> hard shutdown.
>
> A hard shutdown to btrfs itself should not cause anything wrong, that's
> ensured by its mandatory metadata COW.
>
>> So that in itself wouldn't be the problem, I'd think.
>> But the other day I again booted from the old system from its
>> hibernation with the forgotten nvme mounted.
>
> Oh I got it now, it's not after the hibernation immediately, but the
> resume from hibernation, and then some write into the already
> out-of-sync fs caused the problem.
>
>> And that was the killer, I'd say, since a lot of metadata has changed on
>> that btrfs meanwhile.
>
> Yes, I believe that's the case.
>
...
>
> Personally speaking, I never trust hibernation/suspension, although due
> to other ACPI related reasons.
> Now I won't even touch hibernation/suspension at all, just to avoid any
> removable storage corruption.
>
I wonder if it is possible to add resume hook to check that on-disk
state matches in-memory state and go read-only if on-disk state changed.
Checking current generation should be enough to detect it?
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Corrupted btrfs (LUKS), seeking advice
2022-08-10 6:10 ` Andrei Borzenkov
@ 2022-08-10 6:29 ` Qu Wenruo
2022-08-10 7:26 ` Goffredo Baroncelli
0 siblings, 1 reply; 12+ messages in thread
From: Qu Wenruo @ 2022-08-10 6:29 UTC (permalink / raw)
To: Andrei Borzenkov, linux-btrfs; +Cc: wqu
On 2022/8/10 14:10, Andrei Borzenkov wrote:
> On 09.08.2022 14:37, Qu Wenruo wrote:
> ...
>>>
>>> I think what happened is having had mounted the FS twice by accident:
>>> The former system (Mint 19.3/ext4) has been cloned to a USB-stick which
>>> I can boot from.
>>> In one such session I mounted the new btrfs nvme on the old system for
>>> some data exchange.
>>> I put the old system to hibernation but forgot to unmount the nvme prior
>>> to that. :(
>>
>> Hibernation, I'm not sure about the details, but it looks like there
>> were some corruption reports related to that.
>>
>>>
>>> So when booting up the new system from the nvme it was like having had a
>>> hard shutdown.
>>
>> A hard shutdown to btrfs itself should not cause anything wrong, that's
>> ensured by its mandatory metadata COW.
>>
>>> So that in itself wouldn't be the problem, I'd think.
>>> But the other day I again booted from the old system from its
>>> hibernation with the forgotten nvme mounted.
>>
>> Oh I got it now, it's not after the hibernation immediately, but the
>> resume from hibernation, and then some write into the already
>> out-of-sync fs caused the problem.
>>
>>> And that was the killer, I'd say, since a lot of metadata has changed on
>>> that btrfs meanwhile.
>>
>> Yes, I believe that's the case.
>>
> ...
>>
>> Personally speaking, I never trust hibernation/suspension, although due
>> to other ACPI related reasons.
>> Now I won't even touch hibernation/suspension at all, just to avoid any
>> removable storage corruption.
>>
>
> I wonder if it is possible to add resume hook to check that on-disk
> state matches in-memory state and go read-only if on-disk state changed.
AFAIK what we should do is, using hooks to unmount all removable disks,
before entering suspension/hibernation.
> Checking current generation should be enough to detect it?
IMHO fs doesn't really have any way to know we're going into
hibernation/suspension.
If we have such mechanism, then yes it's possible.
Thanks,
Qu
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Corrupted btrfs (LUKS), seeking advice
2022-08-10 6:29 ` Qu Wenruo
@ 2022-08-10 7:26 ` Goffredo Baroncelli
2022-08-11 6:40 ` Qu Wenruo
0 siblings, 1 reply; 12+ messages in thread
From: Goffredo Baroncelli @ 2022-08-10 7:26 UTC (permalink / raw)
To: Qu Wenruo, Andrei Borzenkov, linux-btrfs; +Cc: wqu, Michael Zacherl
On 10/08/2022 08.29, Qu Wenruo wrote:
>
>
> On 2022/8/10 14:10, Andrei Borzenkov wrote:
>> On 09.08.2022 14:37, Qu Wenruo wrote:
>> ...
>>>>
>>>> I think what happened is having had mounted the FS twice by accident:
>>>> The former system (Mint 19.3/ext4) has been cloned to a USB-stick which
>>>> I can boot from.
>>>> In one such session I mounted the new btrfs nvme on the old system for
>>>> some data exchange.
>>>> I put the old system to hibernation but forgot to unmount the nvme prior
>>>> to that. :(
>>>
>>> Hibernation, I'm not sure about the details, but it looks like there
>>> were some corruption reports related to that.
>>>
>>>>
>>>> So when booting up the new system from the nvme it was like having had a
>>>> hard shutdown.
>>>
>>> A hard shutdown to btrfs itself should not cause anything wrong, that's
>>> ensured by its mandatory metadata COW.
>>>
>>>> So that in itself wouldn't be the problem, I'd think.
>>>> But the other day I again booted from the old system from its
>>>> hibernation with the forgotten nvme mounted.
>>>
>>> Oh I got it now, it's not after the hibernation immediately, but the
>>> resume from hibernation, and then some write into the already
>>> out-of-sync fs caused the problem.
>>>
>>>> And that was the killer, I'd say, since a lot of metadata has changed on
>>>> that btrfs meanwhile.
>>>
>>> Yes, I believe that's the case.
>>>
>> ...
>>>
>>> Personally speaking, I never trust hibernation/suspension, although due
>>> to other ACPI related reasons.
>>> Now I won't even touch hibernation/suspension at all, just to avoid any
>>> removable storage corruption.
>>>
>>
>> I wonder if it is possible to add resume hook to check that on-disk
>> state matches in-memory state and go read-only if on-disk state changed.
>
> AFAIK what we should do is, using hooks to unmount all removable disks,
> before entering suspension/hibernation.
I don't think that this is doable, because before the hibernation there are
some file descriptors opened, so a full unmount is not an option.
>
>> Checking current generation should be enough to detect it?
>
> IMHO fs doesn't really have any way to know we're going into
> hibernation/suspension.
>
> If we have such mechanism, then yes it's possible.
I think that the only available hooks are
struct super_operations.freeze_fs struct super_operations.unfreeze_fs
I think a check about a consistency of the transaction id should be appropriate, and then if it fails do a panic.
But the case of Michael seems a "misuse" to care of: check that all the disks have the same transaction ids.
>
> Thanks,
> Qu
--
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Corrupted btrfs (LUKS), seeking advice
2022-08-10 7:26 ` Goffredo Baroncelli
@ 2022-08-11 6:40 ` Qu Wenruo
2022-08-14 13:18 ` Michael Zacherl
0 siblings, 1 reply; 12+ messages in thread
From: Qu Wenruo @ 2022-08-11 6:40 UTC (permalink / raw)
To: kreijack, Andrei Borzenkov, linux-btrfs; +Cc: wqu, Michael Zacherl
On 2022/8/10 15:26, Goffredo Baroncelli wrote:
> On 10/08/2022 08.29, Qu Wenruo wrote:
>>
>>
>> On 2022/8/10 14:10, Andrei Borzenkov wrote:
>>> On 09.08.2022 14:37, Qu Wenruo wrote:
>>> ...
>>>>>
>>>>> I think what happened is having had mounted the FS twice by accident:
>>>>> The former system (Mint 19.3/ext4) has been cloned to a USB-stick
>>>>> which
>>>>> I can boot from.
>>>>> In one such session I mounted the new btrfs nvme on the old system for
>>>>> some data exchange.
>>>>> I put the old system to hibernation but forgot to unmount the nvme
>>>>> prior
>>>>> to that. :(
>>>>
>>>> Hibernation, I'm not sure about the details, but it looks like there
>>>> were some corruption reports related to that.
>>>>
>>>>>
>>>>> So when booting up the new system from the nvme it was like having
>>>>> had a
>>>>> hard shutdown.
>>>>
>>>> A hard shutdown to btrfs itself should not cause anything wrong, that's
>>>> ensured by its mandatory metadata COW.
>>>>
>>>>> So that in itself wouldn't be the problem, I'd think.
>>>>> But the other day I again booted from the old system from its
>>>>> hibernation with the forgotten nvme mounted.
>>>>
>>>> Oh I got it now, it's not after the hibernation immediately, but the
>>>> resume from hibernation, and then some write into the already
>>>> out-of-sync fs caused the problem.
>>>>
>>>>> And that was the killer, I'd say, since a lot of metadata has
>>>>> changed on
>>>>> that btrfs meanwhile.
>>>>
>>>> Yes, I believe that's the case.
>>>>
>>> ...
>>>>
>>>> Personally speaking, I never trust hibernation/suspension, although due
>>>> to other ACPI related reasons.
>>>> Now I won't even touch hibernation/suspension at all, just to avoid any
>>>> removable storage corruption.
>>>>
>>>
>>> I wonder if it is possible to add resume hook to check that on-disk
>>> state matches in-memory state and go read-only if on-disk state changed.
>>
>> AFAIK what we should do is, using hooks to unmount all removable disks,
>> before entering suspension/hibernation.
>
> I don't think that this is doable, because before the hibernation there are
>
> some file descriptors opened, so a full unmount is not an option.
>
>
>>
>>> Checking current generation should be enough to detect it?
>>
>> IMHO fs doesn't really have any way to know we're going into
>> hibernation/suspension.
>>
>> If we have such mechanism, then yes it's possible.
>
> I think that the only available hooks are
>
> struct super_operations.freeze_fs struct
> super_operations.unfreeze_fs
>
> I think a check about a consistency of the transaction id should be
> appropriate, and then if it fails do a panic.
> But the case of Michael seems a "misuse" to care of: check that all the
> disks have the same transaction ids.
I think your idea is pretty to the point.
If we found some obvious unexpected modification, we can mark the FS RO
to minimize the damage (as long as the unexpected modification is still
correct, the fs will be safe).
I just sent a patch based on your idea, feel free to add your
Suggested-by: and other tags:
https://lore.kernel.org/linux-btrfs/41c43c7d6aa25af13391905061e2d203f7853382.1660199812.git.wqu@suse.com/T/#u
Thanks,
Qu
>
>
>>
>> Thanks,
>> Qu
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Corrupted btrfs (LUKS), seeking advice
2022-08-11 6:40 ` Qu Wenruo
@ 2022-08-14 13:18 ` Michael Zacherl
0 siblings, 0 replies; 12+ messages in thread
From: Michael Zacherl @ 2022-08-14 13:18 UTC (permalink / raw)
To: linux-btrfs; +Cc: wqu, Qu Wenruo, kreijack, Andrei Borzenkov
On 11.08.2022, at 08:40, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>> If we have such mechanism, then yes it's possible.
>>
>> I think that the only available hooks are
>>
>> struct super_operations.freeze_fs struct
>> super_operations.unfreeze_fs
>>
>> I think a check about a consistency of the transaction id should be
>> appropriate, and then if it fails do a panic.
>> But the case of Michael seems a "misuse" to care of: check that all the
>> disks have the same transaction ids.
“misuse” indeed, I have a tendency (predisposition?) to exhaust technical possibilities. 🙄
> I think your idea is pretty to the point.
>
> If we found some obvious unexpected modification, we can mark the FS RO
> to minimize the damage (as long as the unexpected modification is still
> correct, the fs will be safe).
I observed this (user POV) behaviour on an old ext4 system:
After waking the hibernating ext4 system again the FS was mounted RO and the OS unusable.
Rebooting dropped into initramfs prompt and an fsck was needed.
Then booting could be continued.
> I just sent a patch based on your idea, feel free to add your
> Suggested-by: and other tags:
> https://lore.kernel.org/linux-btrfs/41c43c7d6aa25af13391905061e2d203f7853382.1660199812.git.wqu@suse.com/T/#u
Thank you very much to you and all for all your efforts - specifically and in general!
Michael.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Corrupted btrfs (LUKS), seeking advice
2022-08-08 13:06 Corrupted btrfs (LUKS), seeking advice Michael Zacherl
2022-08-08 16:57 ` Chris Murphy
@ 2022-08-09 0:22 ` Qu Wenruo
1 sibling, 0 replies; 12+ messages in thread
From: Qu Wenruo @ 2022-08-09 0:22 UTC (permalink / raw)
To: Michael Zacherl, linux-btrfs
On 2022/8/8 21:06, Michael Zacherl wrote:
> Hello,
> on the occasion of retrofitting a 2TB ssd for my old XPS13 9350 I
> decided to give btrfs w/ encryption a try (this was in June).
> Now, by a dumb mistake, I have a corrupted btrfs (LUKS encrypted).
> Since I can't boot from this partition anymore I'm using the distro's
> live system.
> This partition can't be mounted.
>
> # uname -a
> Linux EndeavourOS 5.18.5-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 16 Jun 2022
> 20:40:45 +0000 x86_64 GNU/Linux
>
> # btrfs --version
> btrfs-progs v5.18.1
>
> What I did so far:
>
> # cryptsetup open /dev/nvme0n1p2 luks-test
> Enter passphrase for /dev/nvme0n1p2:
> [worked]
> # mount -o ro,rescue=usebackuproot /dev/mapper/luks-test /mnt
> mount: /mnt: wrong fs type, bad option, bad superblock on
> /dev/mapper/luks-test, missing codepage or helper program, or other error.
> dmesg(1) may have more information after failed mount system call.
>
> dmesg after this mount attempt:
> [ 5179.422225] BTRFS info (device dm-1): flagging fs with big metadata
> feature
> [ 5179.422248] BTRFS info (device dm-1): trying to use backup root at
> mount time
> [ 5179.422255] BTRFS info (device dm-1): using free space tree
> [ 5179.422260] BTRFS info (device dm-1): has skinny extents
> [ 5179.431117] BTRFS error (device dm-1): parent transid verify failed
> on 334692352 wanted 14761 found 14765
> [ 5179.431338] BTRFS error (device dm-1): parent transid verify failed
> on 334692352 wanted 14761 found 14765
> [ 5179.431358] BTRFS error (device dm-1): failed to read block groups: -5
> [ 5179.433358] BTRFS error (device dm-1): open_ctree failed
>
> # btrfs check /dev/mapper/luks-test 2>&1|less -I
> parent transid verify failed on 334692352 wanted 14761 found 14765
> parent transid verify failed on 334692352 wanted 14761 found 14765
> parent transid verify failed on 334692352 wanted 14761 found 14765
> Ignoring transid failure
> [1/7] checking root items
> parent transid verify failed on 334643200 wanted 14761 found 14765
> parent transid verify failed on 334643200 wanted 14761 found 14765
> parent transid verify failed on 334643200 wanted 14761 found 14765
> Ignoring transid failure
> parent transid verify failed on 334659584 wanted 14761 found 14765
> parent transid verify failed on 334659584 wanted 14761 found 14765
> parent transid verify failed on 334659584 wanted 14761 found 14765
> Ignoring transid failure
> parent transid verify failed on 334430208 wanted 6728 found 14763
> parent transid verify failed on 334430208 wanted 6728 found 14763
> parent transid verify failed on 334430208 wanted 6728 found 14763
> Ignoring transid failure
> parent transid verify failed on 334675968 wanted 14761 found 14765
> parent transid verify failed on 334675968 wanted 14761 found 14765
> parent transid verify failed on 334675968 wanted 14761 found 14765
> Ignoring transid failure
> parent transid verify failed on 335216640 wanted 6728 found 14765
> parent transid verify failed on 335216640 wanted 6728 found 14765
> parent transid verify failed on 335216640 wanted 6728 found 14765
> Ignoring transid failure
> parent transid verify failed on 320847872 wanted 14323 found 14763
> parent transid verify failed on 320847872 wanted 14323 found 14763
> parent transid verify failed on 320847872 wanted 14323 found 14763
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=119848960 item=49 parent
> level=1 child bytenr=320847872 child level=1
> ERROR: failed to repair root items: Input/output error
> [2/7] checking extents
> parent transid verify failed on 340246528 wanted 14741 found 14764
> parent transid verify failed on 340246528 wanted 14741 found 14764
> parent transid verify failed on 340246528 wanted 14741 found 14764
> Ignoring transid failure
> [... skipping many lines]
> root 257 inode 1866942 errors 2001, no inode item, link count wrong
> unresolved ref dir 5719 index 9016 namelen 28 name
> SiteSecurityServiceState.txt filetype 1 errors 4, no inode ref
> root 257 inode 1866943 errors 2001, no inode item, link count wrong
> unresolved ref dir 5719 index 9018 namelen 21 name
> AlternateServices.txt filetype 1 errors 4, no inode ref
> root 257 inode 1866989 errors 2001, no inode item, link count wrong
> unresolved ref dir 346 index 16043 namelen 4 name user filetype
> 1 errors 4, no inode ref
> root 257 inode 1866990 errors 2001, no inode item, link count wrong
> unresolved ref dir 1216 index 11701 namelen 46 name
> 0_3_1920_1080_8b47947fd8179de11b12e22fa2a454c8 filetype 1 errors 4, no
> inode ref
> root 257 inode 1866991 errors 2001, no inode item, link count wrong
> unresolved ref dir 5720 index 6961 namelen 16 name
> recovery.jsonlz4 filetype 1 errors 4, no inode ref
> root 257 inode 1866995 errors 2001, no inode item, link count wrong
> unresolved ref dir 6765 index 134 namelen 42 name
> 3647222921wleabcEoxlt-eengsairo.sqlite-wal filetype 1 errors 4, no inode
> ref
> parent transid verify failed on 348258304 wanted 14749 found 14766
> Ignoring transid failure
> parent transid verify failed on 348258304 wanted 14749 found 14766
> Ignoring transid failure
> parent transid verify failed on 348258304 wanted 14749 found 14766
> Ignoring transid failure
> parent transid verify failed on 348258304 wanted 14749 found 14766
> Ignoring transid failure
> ERROR: errors found in fs roots
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/luks-test
> UUID: 2d1dc6b4-84ab-4c64-91a0-669b6228c516
> found 83720974336 bytes used, error(s) found
> total csum bytes: 50467580
> total tree bytes: 266665984
> total fs tree bytes: 186236928
> total extent tree bytes: 17317888
> btree space waste bytes: 40645922
> file data blocks allocated: 87345299456
> referenced 47847440384
> [less: lines 1364572-1364611/1364611 byte 90015202/90015202 (END)
> (press RETURN)]
>
> # btrfs-find-root /dev/mapper/luks-test
> parent transid verify failed on 334692352 wanted 14761 found 14765
> parent transid verify failed on 334692352 wanted 14761 found 14765
> ERROR: failed to read block groups: Input/output error
> Superblock thinks the generation is 14761
> Superblock thinks the level is 0
> Found tree root at 444612608 gen 14761 level 0
> Well block 347160576(gen: 14768 level: 0) seems good, but
> generation/level doesn't match, want gen: 14761 level: 0
> Well block 348192768(gen: 14766 level: 0) seems good, but
> generation/level doesn't match, want gen: 14761 level: 0
> Well block 347865088(gen: 14765 level: 0) seems good, but
> generation/level doesn't match, want gen: 14761 level: 0
> Well block 418840576(gen: 14758 level: 0) seems good, but
> generation/level doesn't match, want gen: 14761 level: 0
> Well block 417120256(gen: 14749 level: 0) seems good, but
> generation/level doesn't match, want gen: 14761 level: 0
> Well block 352256000(gen: 14743 level: 0) seems good, but
> generation/level doesn't match, want gen: 14761 level: 0
> [end]
>
> This is what I found out by reading.
> A fix - if possible - is out of my league now and don't want to poke
> around and make things worse.
> Any chance to get this FS at least mounted for RO?
You can try "mount -o ro,rescue=all", which will skip the block group
item search, but still there will be other corruptions.
I'm more interested in how this happened.
The main point here is, the found transid is newer than the on-disk
transid, which means metadata COW is not working at all.
Are you using unsafe mount options like disabling barriers?
Thanks,
Qu
>
> Thanks a lot, Michael.
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2022-08-14 13:23 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-08-08 13:06 Corrupted btrfs (LUKS), seeking advice Michael Zacherl
2022-08-08 16:57 ` Chris Murphy
2022-08-08 17:33 ` Michael Zacherl
2022-08-09 0:24 ` Qu Wenruo
2022-08-09 11:23 ` Michael Zacherl
2022-08-09 11:37 ` Qu Wenruo
2022-08-10 6:10 ` Andrei Borzenkov
2022-08-10 6:29 ` Qu Wenruo
2022-08-10 7:26 ` Goffredo Baroncelli
2022-08-11 6:40 ` Qu Wenruo
2022-08-14 13:18 ` Michael Zacherl
2022-08-09 0:22 ` Qu Wenruo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).