Why do I need 4 disks for a raid6?

Why do I need 4 disks for a raid6?

[Goswin von Brederlow]

Hi,

I'm wondering why the kernel requires a raid6 to have at least 4
disks (of which at most 2 can be missing). Why not 3 disks?

I know a 1+2 disk raid6 sounds stupid but it has an use case. Let me
demonstrate this with raid5:

Lets say you started with a raid1 when you bought your computer:
mdadm --create -l 1 -n 2 /dev/md9 /dev/sda1 /dev/sdb1
a fewyears of use on /dev/md9 

Now you run out of space and went and bought a 3rd harddisk. What to
do now?
mdadm --stop /dev/md9
mdadm --create -l 5 -n 2 --assume-clean /dev/sda1 /dev/sdb1
mdadm --add /dev/md9 /dev/sdc1
mdadm --grow -n 3 /dev/md9

Voila. A nice raid5 with twice the size of the raid1. And no single
point of failure like degrading the raid1 and copying the data to a
degraded raid5 would have. And the downtime is just a minute to create
the new superblock.



Now for the raid6 case. With only 1 data disk and 2 parity disks all 3
disks should end up with identical data on them. In effect this should
be a 3 disk raid1, a cpu intensive one. Take an existing raid1 with 2
or 3 disks, stop the raid, create a new raid6 ovver it with
--assume-clean, start the raid. After that one can add more disks and
--grow -n 4/5/6/.. the raid6 to a sensible size. Again without going
into degraded mode.


So back to my original question: Why does the kernel require 4 disks
for a raid6 instead of allowing 3?

MfG
        Goswin

PS: Please CC me on replies.

Re: Why do I need 4 disks for a raid6?

[david.geib]

> Now for the raid6 case. With only 1 data disk and 2 parity disks all 3
> disks should end up with identical data on them. In effect this should
> be a 3 disk raid1, a cpu intensive one. Take an existing raid1 with 2
> or 3 disks, stop the raid, create a new raid6 ovver it with
> --assume-clean, start the raid. After that one can add more disks and
> --grow -n 4/5/6/.. the raid6 to a sensible size. Again without going
> into degraded mode.

If this is the only case where it would be useful, wouldn't it be better
to add an option to mdadm --grow specifying a new raid level, if
different? That way you could take your /dev/md9 raid1 and do:

mdadm --add /dev/md9 /dev/sdc1
mdadm --grow -n 3 /dev/md9 --level=5

or

mdadm --add /dev/md9 /dev/sdc1 /dev/sdd1
mdadm --grow -n 4 /dev/md9 --level=6

or convert a raid5 to raid6 in like fashion, for that matter.

Re: Why do I need 4 disks for a raid6?

[Goswin von Brederlow]

david.geib@huskymail.uconn.edu writes:

>> Now for the raid6 case. With only 1 data disk and 2 parity disks all 3
>> disks should end up with identical data on them. In effect this should
>> be a 3 disk raid1, a cpu intensive one. Take an existing raid1 with 2
>> or 3 disks, stop the raid, create a new raid6 ovver it with
>> --assume-clean, start the raid. After that one can add more disks and
>> --grow -n 4/5/6/.. the raid6 to a sensible size. Again without going
>> into degraded mode.
>
> If this is the only case where it would be useful, wouldn't it be better
> to add an option to mdadm --grow specifying a new raid level, if
> different? That way you could take your /dev/md9 raid1 and do:
>
> mdadm --add /dev/md9 /dev/sdc1
> mdadm --grow -n 3 /dev/md9 --level=5
>
> or
>
> mdadm --add /dev/md9 /dev/sdc1 /dev/sdd1
> mdadm --grow -n 4 /dev/md9 --level=6
>
> or convert a raid5 to raid6 in like fashion, for that matter.

That would do 2 things in one, convert the raid level and grow the
stripes.

On the other hand, if 3 disk raid6 where allowed, then this could be
broken down into 2 simple operations for the userspace: 1) change raid
level without restriping, 2) restripe, which we already know how to
do.

To me it just seems easier to teach the kernel to online reload a raid
device with a different raid mode without any changes, covering
conversion of 1<->4, 1<->5 and 1<->6 in both directions, than to
combine the 2 steps.

That would only leave conversion from 4/5 <-> 6 as complicated case.


And hey, maybe I am crazy and do want a 3 disk raid6 next to my 2 disk
raid5, my 1 disk raid10, my 1 disk raid1 and my 1 disk raid0. :)
The limitation in the kernel just seems arbitrary, that is unless one
of the raid algorithms would break down with just 3 disks.

MfG
        Goswin

Re: Why do I need 4 disks for a raid6?

[Andre Noll]

[-- Attachment #1: Type: text/plain, Size: 1029 bytes --]

On 20:12, Goswin von Brederlow wrote:

> I'm wondering why the kernel requires a raid6 to have at least 4
> disks (of which at most 2 can be missing). Why not 3 disks?

Yes, this limitation looks a bit arbitrary. I can not see any reason
why raid6 requires at least four disks. Probably it even works without
any significant changes if some of the checks are being relaxed.

> Now for the raid6 case. With only 1 data disk and 2 parity disks all 3
> disks should end up with identical data on them. In effect this should
> be a 3 disk raid1, a cpu intensive one.

In fact, it wouldn't be much more CPU intensive than raid5 because
the math to "calculate" the Q parity would obviously not involve any
GF multiplications at all.

> So back to my original question: Why does the kernel require 4 disks
> for a raid6 instead of allowing 3?

Dunno. Maybe Dan, Neil or HPA can tell the reason for imposing this
limitation.

Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Why do I need 4 disks for a raid6?

[Andre Noll]

[-- Attachment #1: Type: text/plain, Size: 528 bytes --]

On 08:34, Ruslan Sivak wrote:

> I would guess the reason is that it doesn't make sense.  As mentioned, if
> you are going to create a 3 disk raid 6, it's essentially a raid1 over 3
> disks, at which point you are better off with the raid-1.  I don't think
> there's a raid controller that would let you set something like this up, I
> don't see why the softraid should.

Well, Goswin mentioned some pretty good reasons I think.

Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Why do I need 4 disks for a raid6?

[Goswin von Brederlow]

Andre Noll <maan@systemlinux.org> writes:

> On 08:34, Ruslan Sivak wrote:
>
>> I would guess the reason is that it doesn't make sense.  As mentioned, if
>> you are going to create a 3 disk raid 6, it's essentially a raid1 over 3
>> disks, at which point you are better off with the raid-1.  I don't think
>> there's a raid controller that would let you set something like this up, I
>> don't see why the softraid should.
>
> Well, Goswin mentioned some pretty good reasons I think.
>
> Andre
> -- 
> The only person who always got his work done by Friday was Robinson Crusoe

Raid6 is the only level that requires such "sanity".

Here is another scenario: Say you have a 2 disk raid1 and now want to
switch to 5 disk raid6. No problem, add 3 new disks, set up 1+2 disk
raid6, pvmove the data (don't you love LVM?), stop the raid1, grow the
raid6 to 5 disks.

It is clear that 1+2 disk raid6 only makes sense as a transitory step
but one that is usefull.

MfG
        Goswin

Re: Why do I need 4 disks for a raid6?

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 2000 bytes --]

On Wed Mar 18, 2009 at 03:08:56PM +0100, Goswin von Brederlow wrote:

> Andre Noll <maan@systemlinux.org> writes:
> 
> > On 08:34, Ruslan Sivak wrote:
> >
> >> I would guess the reason is that it doesn't make sense.  As mentioned, if
> >> you are going to create a 3 disk raid 6, it's essentially a raid1 over 3
> >> disks, at which point you are better off with the raid-1.  I don't think
> >> there's a raid controller that would let you set something like this up, I
> >> don't see why the softraid should.
> >
> > Well, Goswin mentioned some pretty good reasons I think.
> >
> 
> Raid6 is the only level that requires such "sanity".
> 
> Here is another scenario: Say you have a 2 disk raid1 and now want to
> switch to 5 disk raid6. No problem, add 3 new disks, set up 1+2 disk
> raid6, pvmove the data (don't you love LVM?), stop the raid1, grow the
> raid6 to 5 disks.
> 
> It is clear that 1+2 disk raid6 only makes sense as a transitory step
> but one that is usefull.
> 
You could do this via the new RAID5-RAID6 migration. Either:
    - recreate the two disk RAID1 as RAID5 (assume clean)
    - add 2 disks and grow the array
    - add the fifth disk and convert to RAID6
or
    - create a new 3-disk RAID5 array
    - copy/pvmove the data
    - add a fourth disk and grow the array
    - add the fifth disk and convert to RAID6
or even (not 100% sure this'll work until the RAID6 restriping work is
done):
    - create a new 3-disk RAID5 array
    - copy/pvmove the data
    - add a fourth disk and convert to RAID6
    - add the fifth disk and grow the array

So the 3-disk RAID6 isn't essential, though if there's no particular
reason to prohibit it then it might as well be allowed as an option.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Why do I need 4 disks for a raid6?

[david.geib]

>> If this is the only case where it would be useful, wouldn't it be better
>> to add an option to mdadm --grow specifying a new raid level, if
>> different? That way you could take your /dev/md9 raid1 and do:
>>
>> mdadm --add /dev/md9 /dev/sdc1
>> mdadm --grow -n 3 /dev/md9 --level=5
>>
>> or
>>
>> mdadm --add /dev/md9 /dev/sdc1 /dev/sdd1
>> mdadm --grow -n 4 /dev/md9 --level=6
>>
>> or convert a raid5 to raid6 in like fashion, for that matter.

> That would do 2 things in one, convert the raid level and grow the
> stripes.
>
> On the other hand, if 3 disk raid6 where allowed, then this could be
> broken down into 2 simple operations for the userspace: 1) change raid
> level without restriping, 2) restripe, which we already know how to
> do.
>
> To me it just seems easier to teach the kernel to online reload a raid
> device with a different raid mode without any changes, covering
> conversion of 1<->4, 1<->5 and 1<->6 in both directions, than to
> combine the 2 steps.
>
> That would only leave conversion from 4/5 <-> 6 as complicated case.

That's really what I'm getting at. In the 4/5 <-> 6 case the level
conversion would require a reshape and either changing the number of
devices or shrinking the array. This implies the need for the userspace to
convey to the kernel both the level conversion and the new number of
devices at the same time. Once that is possible, why not use it for 1 <->
6 and all the others too?

> And hey, maybe I am crazy and do want a 3 disk raid6 next to my 2 disk
> raid5, my 1 disk raid10, my 1 disk raid1 and my 1 disk raid0. :)
> The limitation in the kernel just seems arbitrary, that is unless one
> of the raid algorithms would break down with just 3 disks.

I don't disagree that if a three disk raid6 is possible in theory then it
might as well be supported. I'm merely suggesting that there ought to be a
better way to do level conversion than creating a new array on top of the
old one.

Re: Why do I need 4 disks for a raid6?

[Goswin von Brederlow]

Robin Hill <robin@robinhill.me.uk> writes:

> You could do this via the new RAID5-RAID6 migration. Either:

Oh, someone is working on that. Didn't know that. That certainly makes
3 disk raid6 less important as one can go via raid5 then.

But still. If it is only that one line in the kernel that checks < 4
instead of < 3 then ...

MfG
        Goswin

Re: Why do I need 4 disks for a raid6?

[Neil Brown]

On Tuesday March 17, goswin-v-b@web.de wrote:
> Hi,
> 
> I'm wondering why the kernel requires a raid6 to have at least 4
> disks (of which at most 2 can be missing). Why not 3 disks?
> 
> I know a 1+2 disk raid6 sounds stupid but it has an use case. Let me
> demonstrate this with raid5:
> 
> Lets say you started with a raid1 when you bought your computer:
> mdadm --create -l 1 -n 2 /dev/md9 /dev/sda1 /dev/sdb1
> a fewyears of use on /dev/md9 
> 
> Now you run out of space and went and bought a 3rd harddisk. What to
> do now?
> mdadm --stop /dev/md9
> mdadm --create -l 5 -n 2 --assume-clean /dev/sda1 /dev/sdb1
> mdadm --add /dev/md9 /dev/sdc1
> mdadm --grow -n 3 /dev/md9
> 
> Voila. A nice raid5 with twice the size of the raid1. And no single
> point of failure like degrading the raid1 and copying the data to a
> degraded raid5 would have. And the downtime is just a minute to create
> the new superblock.

In 2.6.30 and later you wont even need the downtime.  Just
  echo raid5 > /sys/block/md9/md/level
in place of the "stop" and "create" steps.

> 
> 
> 
> Now for the raid6 case. With only 1 data disk and 2 parity disks all 3
> disks should end up with identical data on them. In effect this should
> be a 3 disk raid1, a cpu intensive one. Take an existing raid1 with 2
> or 3 disks, stop the raid, create a new raid6 ovver it with
> --assume-clean, start the raid. After that one can add more disks and
> --grow -n 4/5/6/.. the raid6 to a sensible size. Again without going
> into degraded mode.
> 
> 
> So back to my original question: Why does the kernel require 4 disks
> for a raid6 instead of allowing 3?

I have occasionally wondered that.  But I didn't write that code and
never saw a need to change it.  As Andre says, it is very likely that
just relaxing the restriction will allow it to "just work".

Maybe that will happen in 2.6.31...

NeilBrown

Re: Why do I need 4 disks for a raid6?

[Goswin von Brederlow]

Neil Brown <neilb@suse.de> writes:

> In 2.6.30 and later you wont even need the downtime.  Just
>   echo raid5 > /sys/block/md9/md/level
> in place of the "stop" and "create" steps.

Hurray. It is great to see the raid driver improving all the
time. Great work all of you.
 
>> Now for the raid6 case. With only 1 data disk and 2 parity disks all 3
>> disks should end up with identical data on them. In effect this should
>> be a 3 disk raid1, a cpu intensive one. Take an existing raid1 with 2
>> or 3 disks, stop the raid, create a new raid6 ovver it with
>> --assume-clean, start the raid. After that one can add more disks and
>> --grow -n 4/5/6/.. the raid6 to a sensible size. Again without going
>> into degraded mode.
>> 
>> 
>> So back to my original question: Why does the kernel require 4 disks
>> for a raid6 instead of allowing 3?
>
> I have occasionally wondered that.  But I didn't write that code and
> never saw a need to change it.  As Andre says, it is very likely that
> just relaxing the restriction will allow it to "just work".
>
> Maybe that will happen in 2.6.31...
>
> NeilBrown

Time for try&error. I only checked one raid6 algorithm but that seemed
to be fine with 3 disks.

MfG
        Goswin

Re: Why do I need 4 disks for a raid6?

[Nifty Fedora Mitch]

> On Tuesday March 17, goswin-v-b@web.de wrote:
> > Hi,
> > 
> > I'm wondering why the kernel requires a raid6 to have at least 4
> > disks (of which at most 2 can be missing). Why not 3 disks?

I should think that this is the 'defination' of a raid6.
If you build a raid6 resource it should be raid6 and
as such be able to tolerate the loss of two disks
and all the other raid6 properties.

If you build a raidN with three disks that can tolerate the
loss of one disk it is not raid6 but raid5 and should be called
by the correct descriptive name.

The importance of this strictness surfaces in documentation.   
Both to the system admin working on the system and also
documentation from the system admin to his management that
is comparing prices and matching features.   i.e. "Bob's raid6 
is 25% less costly than Fred's".   Truth in advertising comes
to play.

Hidden in this thread is an interesting notion of migration from a
"lesser" raid to a more durable raid over time.   It might make sense to
facilitate limited tools to this end.   Even for those sites that wish to stage
the construction of a large raid install or to stagger the poweron hours
and date codes of drives on the common notion that batches of drives
fail together.  Or perhaps a "raid6 ready" raid5 that gets populated with
the absent drive on the first statistical sniff of an error outside of
the norm.  Or the arrival of warrenty expiration....

Such a "raidN ready" setup does confuse the notion of operational health
for any monitoring tool and is likely a very bad idea for that reason alone.

This 2001 URL is interesting in the comments about how unfortunate the
choice of the world "level" was.
   http://www.pcguide.com/ref/hdd/perf/raid/levels/index.htm
also interesting is the coment about a vendor being sloppy
with the technical language.


-- 
	T o m  M i t c h e l l 
	Found me a new hat, now what?

Re: Why do I need 4 disks for a raid6?

[Goswin von Brederlow]

Nifty Fedora Mitch <niftyfedora@niftyegg.com> writes:

>> On Tuesday March 17, goswin-v-b@web.de wrote:
>> > Hi,
>> > 
>> > I'm wondering why the kernel requires a raid6 to have at least 4
>> > disks (of which at most 2 can be missing). Why not 3 disks?
>
> I should think that this is the 'defination' of a raid6.
> If you build a raid6 resource it should be raid6 and
> as such be able to tolerate the loss of two disks
> and all the other raid6 properties.

Actually the definition I read was that raid6 is like raid5 but
allowing for 2 or more disks to fail without loss. So a 15 disks raid
with 10 data blocks and 5 parity blocks per stripe would also be raid6.

> If you build a raidN with three disks that can tolerate the
> loss of one disk it is not raid6 but raid5 and should be called
> by the correct descriptive name.

I was talking about a raid6 with three disks that can tolerate the
loss of *two* disks. I know that effectively that would be just like a
raid1 with 3 disks but that was exactly the point. To change a 3 disk
raid1 into a 3 disk raid6 and then grow it.

> The importance of this strictness surfaces in documentation.   
> Both to the system admin working on the system and also
> documentation from the system admin to his management that
> is comparing prices and matching features.   i.e. "Bob's raid6 
> is 25% less costly than Fred's".   Truth in advertising comes
> to play.
>
> Hidden in this thread is an interesting notion of migration from a
> "lesser" raid to a more durable raid over time.   It might make sense to
> facilitate limited tools to this end.   Even for those sites that wish to stage
> the construction of a large raid install or to stagger the poweron hours
> and date codes of drives on the common notion that batches of drives
> fail together.  Or perhaps a "raid6 ready" raid5 that gets populated with
> the absent drive on the first statistical sniff of an error outside of
> the norm.  Or the arrival of warrenty expiration....

That would be a raid6 with 4 disks of which one is missing. You can
already do that.

> Such a "raidN ready" setup does confuse the notion of operational health
> for any monitoring tool and is likely a very bad idea for that reason alone.
>
> This 2001 URL is interesting in the comments about how unfortunate the
> choice of the world "level" was.
>    http://www.pcguide.com/ref/hdd/perf/raid/levels/index.htm
> also interesting is the coment about a vendor being sloppy
> with the technical language.

Verry badly advertising polluted url and confusing imho.

http://en.wikipedia.org/wiki/RAID

MfG
        Goswin

Re: Why do I need 4 disks for a raid6?

[H. Peter Anvin]

Andre Noll wrote:
> 
>> So back to my original question: Why does the kernel require 4 disks
>> for a raid6 instead of allowing 3?
> 
> Dunno. Maybe Dan, Neil or HPA can tell the reason for imposing this
> limitation.
> 

It's very simple: it avoids an ugly corner case in the RAID-6
computation code.  Rather than inserting special code (and verifying it,
etc.) to deal with the 3-disk RAID-6 degenerate case, it was easier to
just not permit it.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: Why do I need 4 disks for a raid6?

[H. Peter Anvin]

Goswin von Brederlow wrote:
> 
> Actually the definition I read was that raid6 is like raid5 but
> allowing for 2 or more disks to fail without loss. So a 15 disks raid
> with 10 data blocks and 5 parity blocks per stripe would also be raid6.
> 

2 or more *arbitrary* disks.  You can't, say, have two RAID-5s and call
the combination a RAID-6, because they're not all there.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: Why do I need 4 disks for a raid6?

[Andre Noll]

[-- Attachment #1: Type: text/plain, Size: 1149 bytes --]

On 09:08, H. Peter Anvin wrote:
> Andre Noll wrote:
> > 
> >> So back to my original question: Why does the kernel require 4 disks
> >> for a raid6 instead of allowing 3?
> > 
> > Dunno. Maybe Dan, Neil or HPA can tell the reason for imposing this
> > limitation.
> > 
> 
> It's very simple: it avoids an ugly corner case in the RAID-6
> computation code.  Rather than inserting special code (and verifying it,
> etc.) to deal with the 3-disk RAID-6 degenerate case, it was easier to
> just not permit it.

Out of curiosity, why is any special-casing needed for the 3-disk case?

As

	P = D_0 + ... + D_{n-1}

coincides with

	Q = g^0D_0 + ... + g^{n-1}D_{n-1}

in the case n=1 (because only the first addend is present), we have

	D_0 = P = Q

i.e. raid6 with three disks is the same as raid1. If we lose D_0 and P,
the general formula for reconstruction, i.e.

	D_x = g^{-x}(Q + Q_x)

still works for n=1: It reduces to D_0 = Q since x=0 and Q_0 = 0.
So I don't see why special-casing the n=1 case is necessary.

Regards
Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Why do I need 4 disks for a raid6?

[Goswin von Brederlow]

"H. Peter Anvin" <hpa@zytor.com> writes:

> Andre Noll wrote:
>> 
>>> So back to my original question: Why does the kernel require 4 disks
>>> for a raid6 instead of allowing 3?
>> 
>> Dunno. Maybe Dan, Neil or HPA can tell the reason for imposing this
>> limitation.
>> 
>
> It's very simple: it avoids an ugly corner case in the RAID-6
> computation code.  Rather than inserting special code (and verifying it,
> etc.) to deal with the 3-disk RAID-6 degenerate case, it was easier to
> just not permit it.
>
> 	-hpa

Could you point me at the file/line where the corner case is?

MfG
        Goswin

Re: Why do I need 4 disks for a raid6?

[H. Peter Anvin]

Goswin von Brederlow wrote:
> 
> Could you point me at the file/line where the corner case is?
> 

No, am I'm not even sure if it's there anymore.  It's been a long time.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: Why do I need 4 disks for a raid6?

[H. Peter Anvin]

Andre Noll wrote:
> 
> Out of curiosity, why is any special-casing needed for the 3-disk case?
> 

As I remember it, at least one of the optimized calculation loops didn't 
like the case of only one data drive.

	-hpa