Re: MD RAID 5/6 vs BTRFS RAID 5/6

From: Graham Cobb <g.btrfs@cobb.uk.net>
To: Chris Murphy <lists@colorremedies.com>,
	Btrfs BTRFS <linux-btrfs@vger.kernel.org>
Cc: Qu Wenruo <quwenruo.btrfs@gmx.com>
Subject: Re: MD RAID 5/6 vs BTRFS RAID 5/6
Date: Thu, 17 Oct 2019 19:23:40 +0100	[thread overview]
Message-ID: <be2aa15e-0539-219f-c2d6-fdb01297351e@cobb.uk.net> (raw)
In-Reply-To: <CAJCQCtR=NQd6uovvAhuTdxRNJtnMFDtkTma9u8-Ep9Nq+YQY=A@mail.gmail.com>

On 17/10/2019 16:57, Chris Murphy wrote:
> On Wed, Oct 16, 2019 at 10:07 PM Jon Ander MB <jonandermonleon@gmail.com> wrote:
>>
>> It would be interesting to know the pros and cons of this setup that
>> you are suggesting vs zfs.
>> +zfs detects and corrects bitrot (
>> http://www.zfsnas.com/2015/05/24/testing-bit-rot/ )
>> +zfs has working raid56
>> -modules out of kernel for license incompatibilities (a big minus)
>>
>> BTRFS can detect bitrot but... are we sure it can fix it? (can't seem
>> to find any conclusive doc about it right now)
> 
> Yes. Active fixups with scrub since 3.19. Passive fixups since 4.12.

Presumably this is dependent on checksums? So neither detection nor
fixup happen for NOCOW files? Even a scrub won't notice because scrub
doesn't attempt to compare both copies unless the first copy has a bad
checksum -- is that correct?

> 
>> I'm one of those that is waiting for the write hole bug to be fixed in
>> order to use raid5 on my home setup. It's a shame it's taking so long.
> 
> For what it's worth, the write hole is considered to be rare.
> https://lwn.net/Articles/665299/
> 
> Further, the write hole means a) parity is corrupt or stale compared
> to data stripe elements which is caused by a crash or powerloss during
> writes, and b) subsequently there is a missing device or bad sector in
> the same stripe as the corrupt/stale parity stripe element. The effect
> of b) is that reconstruction from parity is necessary, and the effect
> of a) is that it's reconstructed incorrectly, thus corruption. But
> Btrfs detects this corruption, whether it's metadata or data. The
> corruption isn't propagated in any case. But it makes the filesystem
> fragile if this happens with metadata. Any parity stripe element
> staleness likely results in significantly bad reconstruction in this
> case, and just can't be worked around, even btrfs check probably can't
> fix it. If the write hole problem happens with data block group, then
> EIO. But the good news is that this isn't going to result in silent
> data or file system metadata corruption. For sure you'll know about
> it.

If I understand correctly, metadata always has checksums so that is true
for filesystem structure. But for no-checksum files (such as nocow
files) the corruption will be silent, won't it?

Graham