Re: Filesystem corruption on RAID1

[Chris Murphy <lists@colorremedies.com>]

On Mon, Aug 21, 2017 at 2:37 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Sun, 20 Aug 2017, Chris Murphy wrote:
>
>> Since md doesn't read from both mirrors, it's possible there's a read from
>> a non-corrupt drive, which presents good information to fsck, which then
>> sees no reason to fix anything in that block; but the other mirror does have
>> corruption which thus goes undetected.
>
>
> That was exactly what I wrote.
>
>> One way of dealing with it is to scrub (repair) so they both have the same
>> information to hand over to fsck. Fixups then get replicated to disks by md.
>
>
> Yes, it is, but that would require a full repair before doing fsck. That
> seems excessive because that will take hours on larger drives.

Hence we have ZFS and Btrfs and dm-integrity to unambiguously identify
corruption and prevent it from escaping to higher levels.

That you have multiple incongruencies with fs metadata, there's a good
chance some data is also affected. Data is a much bigger percentage.

Might as well bite the bullet and scrub the whole thing.


>
>> Another way is to split the mirror (make one device faulty), and then
>> fix the remaining drive (now degraded). If that goes well, the 2nd
>> device can be re-added. Here's a caveat thought: how it resync's will
>> depend on the write-intent bitmap being present. I have no idea if
>> write-intent bitmaps on two drives can get out of sync and what the
>> ensuing behavior is, but I'd like to think md will discover the fixed
>> drive event count is higher than the re-added one, and if necessary
>> does a full resync, rather than possibly re-introducing any
>> corruption.
>
>
> This doesn't solve the problem because it doesn't check if the second mirror
> is out of sync with the first one, because it'll only detect writes to the
> degraded array and sync those. It doesn't fix the "fsck read the block and
> it was fine, but on the second drive it's not fine".
>
> In that case fsck would have to be modified to write all blocks it read to
> make them dirty, so they're sync:ed.

OK so you have a corrupt underlying storage stack for possibly unknown
reasons, and you're just going to take a chance and overwrite the
entire file system. Seems like a bad hack to me, but I'd love to know
what the ext4 and XFS devs think about it.

The rule has always been get lower levels healthy first. Two mirrors
that have the same even count but are not block identical is a broken
array.


-- 
Chris Murphy