Re: RAID6 dead on the water after Controller failure

[Phil Turmel <philip@turmel.org>]

On 02/15/2014 06:23 PM, Jon Nelson wrote:
> On Sat, Feb 15, 2014 at 5:04 PM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>> On Sat, 15 Feb 2014, Jon Nelson wrote:
>>
>>> Out of 12 drives, I thought RAID6 only offered a total of *2* failed
>>> devices. It seems to me that you have 7 devices in sync and 4 *almost* in
>>> sync. It's this "almost" part that has me confused. How can the raid run if
>>> the event count doesn't match? Wouldn't at least 10 out of 12 drives have to
>>> have the same event count to avoid data loss?
>>
>>
>> Correct. When you use --assemble --force you're basically telling mdadm "I
>> know what I'm doing and I'll take the risk of data loss or corruption". If
>> you assemble in with a kicked drive that was kicked long ago that has a
>> really far off event count, you can really really screw things up.
>>
>> Unless you use --force, mdadm won't assemble an array where the event count
>> doesn't match up.
> 
> Aha. So if you ran a check in this case, it would find some number of
> blocks that don't match up. What does MD do in that case?

Nothing other than reporting it in the mismatch_cnt.  If you then
perform a "repair" scrub, it will regenerate P&Q from the data blocks,
period.

If you think about it, five of the seven events are easily explained:
four dropped drives, plus cancellation of the rebuild onto what was then
/dev/sdh1.  So the opportunity for other corruption was small.  This is
precisely the scenario where forced assembly makes sense.  The drives
were failed for reasons other than actual drive failure.  The use of
--force tells mdadm that those drives aren't really bad.  But I left out
the partially rebuilt drive because it really couldn't be trusted.

HTH,

Phil