From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chris Murphy Subject: Re: Filesystem corruption on RAID1 Date: Sun, 20 Aug 2017 17:22:58 -0600 Message-ID: References: <20170713214856.4a5c8778@natsu> <592f19bf608e9a959f9445f7f25c5dad@assyoma.it> <770b09d3-cff6-b6b2-0a51-5d11e8bac7e9@thelounge.net> <9eea45ddc0f80f4f4e238b5c2527a1fa@assyoma.it> <7ca98351facca6e3668d3271422e1376@assyoma.it> <5995D377.9080100@youngman.org.uk> <83f4572f09e7fbab9d4e6de4a5257232@assyoma.it> <59961DD7.3060208@youngman.org.uk> <784bec391a00b9e074744f31901df636@assyoma.it> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Return-path: In-Reply-To: Sender: linux-raid-owner@vger.kernel.org To: Mikael Abrahamsson Cc: Gionatan Danti , Roger Heflin , Wols Lists , Reindl Harald , Roman Mamedov , Linux RAID List-Id: linux-raid.ids On Sun, Aug 20, 2017 at 1:14 AM, Mikael Abrahamsson wrote: > After a non-clean poweroff and possible mismatch now between the RAID1 > drives, and now fsck runs. It reads from the drives and fixes problem. > However because the RAID1 drives contain different information, some of the > errors are not fixed. Next time anything comes along, it might read from a > different drive than what fsck read from, and now we have corruption. The fsck has no idea this is two drives, it things it's one and does an overwrite of whatever (virtual) blocks contain file system metadata needing repair. Then md should take each fsck write, and duplicate it (for 2 way mirror) and push those writes to each real physical device. Since md doesn't read from both mirrors, it's possible there's a read from a non-corrupt drive, which presents good information to fsck, which then sees no reason to fix anything in that block; but the other mirror does have corruption which thus goes undetected. One way of dealing with it is to scrub (repair) so they both have the same information to hand over to fsck. Fixups then get replicated to disks by md. Another way is to split the mirror (make one device faulty), and then fix the remaining drive (now degraded). If that goes well, the 2nd device can be re-added. Here's a caveat thought: how it resync's will depend on the write-intent bitmap being present. I have no idea if write-intent bitmaps on two drives can get out of sync and what the ensuing behavior is, but I'd like to think md will discover the fixed drive event count is higher than the re-added one, and if necessary does a full resync, rather than possibly re-introducing any corruption. -- Chris Murphy