About the state of Btrfs RAID 5/6

About the state of Btrfs RAID 5/6

[Neal Gompa]

Hey all,

It keeps coming up over and over: "Btrfs RAID 5/6 is bad, therefore
Btrfs is bad". I know in practice that most people don't use the
raid56 modes for various reasons that are unrelated to this, but it's
frustrating and annoying that it keeps coming up.

I know that Zygo recently posted about how to use RAID 5/6
successfully(-ish)[1], and previously David Sterba tried to work on
this[2]. The result of that was the new raid1cX modes, but we still
have the raid56 modes.

What's holding back Btrfs raid56 from having its status[3] updated
from "Unstable" to "mostly OK" or even "OK"? I know that Synology NAS
devices use Btrfs and support raid56 using mdraid, why not fix the
native RAID features so that layer can be dropped?

[1]: https://lore.kernel.org/linux-btrfs/20200627032414.GX10769@hungrycats.org/
[2]: https://hackweek.suse.com/17/projects/do-something-about-btrfs-and-raid56
[3]: https://btrfs.wiki.kernel.org/index.php/Status

-- 
真実はいつも一つ！/ Always, there's only one truth!

Re: About the state of Btrfs RAID 5/6

[Josef Bacik]

On 9/2/20 11:10 AM, Neal Gompa wrote:
> Hey all,
> 
> It keeps coming up over and over: "Btrfs RAID 5/6 is bad, therefore
> Btrfs is bad". I know in practice that most people don't use the
> raid56 modes for various reasons that are unrelated to this, but it's
> frustrating and annoying that it keeps coming up.
> 
> I know that Zygo recently posted about how to use RAID 5/6
> successfully(-ish)[1], and previously David Sterba tried to work on
> this[2]. The result of that was the new raid1cX modes, but we still
> have the raid56 modes.
> 
> What's holding back Btrfs raid56 from having its status[3] updated
> from "Unstable" to "mostly OK" or even "OK"? I know that Synology NAS
> devices use Btrfs and support raid56 using mdraid, why not fix the
> native RAID features so that layer can be dropped?
> 

Honestly there's just a lot of things that need to be looked at right now.  Poor 
Zygo is waiting on the delayed refs flushing work I did months ago because I 
still haven't gotten back to fixing relocation.

At the moment my core priority is getting all the normal parts of the file 
system working properly, and I've got probably 18 months of hell left before I 
feel like I'll be able to look up and address the multi-disk aspect of btrfs. 
RAID0/1/cX modes all work well and are well understood, and that feels "good 
enough" to me for right now.  I know Johannes is working on something magical, 
but IDK what it is or how far out it is.  Thanks,

Josef

Re: About the state of Btrfs RAID 5/6

[Neal Gompa]

On Wed, Sep 2, 2020 at 1:37 PM Josef Bacik <josef@toxicpanda.com> wrote:
>
> On 9/2/20 11:10 AM, Neal Gompa wrote:
> > Hey all,
> >
> > It keeps coming up over and over: "Btrfs RAID 5/6 is bad, therefore
> > Btrfs is bad". I know in practice that most people don't use the
> > raid56 modes for various reasons that are unrelated to this, but it's
> > frustrating and annoying that it keeps coming up.
> >
> > I know that Zygo recently posted about how to use RAID 5/6
> > successfully(-ish)[1], and previously David Sterba tried to work on
> > this[2]. The result of that was the new raid1cX modes, but we still
> > have the raid56 modes.
> >
> > What's holding back Btrfs raid56 from having its status[3] updated
> > from "Unstable" to "mostly OK" or even "OK"? I know that Synology NAS
> > devices use Btrfs and support raid56 using mdraid, why not fix the
> > native RAID features so that layer can be dropped?
> >
>
> Honestly there's just a lot of things that need to be looked at right now.  Poor
> Zygo is waiting on the delayed refs flushing work I did months ago because I
> still haven't gotten back to fixing relocation.
>
> At the moment my core priority is getting all the normal parts of the file
> system working properly, and I've got probably 18 months of hell left before I
> feel like I'll be able to look up and address the multi-disk aspect of btrfs.
> RAID0/1/cX modes all work well and are well understood, and that feels "good
> enough" to me for right now.  I know Johannes is working on something magical,
> but IDK what it is or how far out it is.  Thanks,
>

That's fair. I just want to have a better idea of what's in the
pipeline here so I can figure out my messaging and roadmap in Fedora.

Downstream in Fedora, we have a tracking item for documenting that
raid56 should not be used at this time[1], and I just want to collect
information for the purpose of providing a more complete picture as
part of the documentation. Today, Anaconda will not expose the ability
to create raid56 btrfs anyway, so people using that configuration will
be relatively rare. But having a good story about it is something I'd
like to have, if possible.

[1]: https://pagure.io/fedora-btrfs/project/issue/13

-- 
真実はいつも一つ！/ Always, there's only one truth!

Re: About the state of Btrfs RAID 5/6

[Johannes Thumshirn]

On 02/09/2020 19:37, Josef Bacik wrote:
> I know Johannes is working on something magical, 
> but IDK what it is or how far out it is.

That something is still slide-ware. I hopefully get back 
to really working on it soonish.

So please don't expect patches within the next say 12 months.

Re: About the state of Btrfs RAID 5/6

[Damien Le Moal]

On 2020/09/03 16:34, Johannes Thumshirn wrote:
> On 02/09/2020 19:37, Josef Bacik wrote:
>> I know Johannes is working on something magical, 
>> but IDK what it is or how far out it is.
> 
> That something is still slide-ware. I hopefully get back 
> to really working on it soonish.
> 
> So please don't expect patches within the next say 12 months.

Come on... You can do better than that :)

Joke aside, once we are past the zoned block device support (new version coming
this week), we will accelerate that work.


-- 
Damien Le Moal
Western Digital Research

Re: About the state of Btrfs RAID 5/6

[Josef Bacik]

On 9/3/20 3:34 AM, Johannes Thumshirn wrote:
> On 02/09/2020 19:37, Josef Bacik wrote:
>> I know Johannes is working on something magical,
>> but IDK what it is or how far out it is.
> 
> That something is still slide-ware. I hopefully get back
> to really working on it soonish.
> 
> So please don't expect patches within the next say 12 months.
> 

Have you seen my branch graveyard of things I still need to get back to? 
Clearly we're not committing to anything here ;).  Thanks,

Josef

Re: About the state of Btrfs RAID 5/6

[Neal Gompa]

On Thu, Sep 3, 2020 at 4:50 AM Damien Le Moal <Damien.LeMoal@wdc.com> wrote:
>
> On 2020/09/03 16:34, Johannes Thumshirn wrote:
> > On 02/09/2020 19:37, Josef Bacik wrote:
> >> I know Johannes is working on something magical,
> >> but IDK what it is or how far out it is.
> >
> > That something is still slide-ware. I hopefully get back
> > to really working on it soonish.
> >
> > So please don't expect patches within the next say 12 months.
>
> Come on... You can do better than that :)
>
> Joke aside, once we are past the zoned block device support (new version coming
> this week), we will accelerate that work.
>

Now that we've got ZBD support landing with Linux 5.12 for Btrfs, any
chance this might be looked at anytime soon?



-- 
真実はいつも一つ！/ Always, there's only one truth!

Re: About the state of Btrfs RAID 5/6

[Damien Le Moal]

On 2021/04/21 7:33, Neal Gompa wrote:
> On Thu, Sep 3, 2020 at 4:50 AM Damien Le Moal <Damien.LeMoal@wdc.com> wrote:
>>
>> On 2020/09/03 16:34, Johannes Thumshirn wrote:
>>> On 02/09/2020 19:37, Josef Bacik wrote:
>>>> I know Johannes is working on something magical,
>>>> but IDK what it is or how far out it is.
>>>
>>> That something is still slide-ware. I hopefully get back
>>> to really working on it soonish.
>>>
>>> So please don't expect patches within the next say 12 months.
>>
>> Come on... You can do better than that :)
>>
>> Joke aside, once we are past the zoned block device support (new version coming
>> this week), we will accelerate that work.
>>
> 
> Now that we've got ZBD support landing with Linux 5.12 for Btrfs, any
> chance this might be looked at anytime soon?

Working on it. But this is not a small change so it will take a while for a
patch series to be posted.



-- 
Damien Le Moal
Western Digital Research