swidth in RAID

swidth in RAID

[aurfalien]

Hi,

I understand swidth should = #data disks.

And the docs say for RAID 6 of 8 disks, that means 6.

But parity is distributed and you actually have 8 disks/spindles working for you and a bit of parity on each.

So shouldn't swidth equal disks in raid when its concerning distributed parity raid?

- aurf
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: swidth in RAID

[Peter Grandi]

> I understand swidth should = #data disks.  And the docs say
> for RAID 6 of 8 disks, that means 6. [ ... ] 8 disks/spindles
> working for you and a bit of parity on each. So shouldn't
> swidth equal disks in raid when its concerning distributed
> parity raid?

The main goal is trying to the reduce the probability of
read-modify-write.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: swidth in RAID

[Stan Hoeppner]

On 6/30/2013 1:43 PM, aurfalien wrote:

> I understand swidth should = #data disks.

No.  "swidth" is a byte value specifying the number of 512 byte blocks
in the data stripe.

"sw" is #data disks.

> And the docs say for RAID 6 of 8 disks, that means 6.
> 
> But parity is distributed and you actually have 8 disks/spindles working for you and a bit of parity on each.
> 
> So shouldn't swidth equal disks in raid when its concerning distributed parity raid?

No.  Lets try visual aids.

Set 8 coffee cups (disk drives) on a table.  Grab a bag of m&m's.
Separate 24 blues (data) and 8 reds (parity).

Drop a blue m&m in cups 1-6 and a red into 7-8.  You just wrote one RAID
stripe.  Now drop a blue into cups 3-8 and a red in 1-2.  Your second
write, this time rotating two cups (drives) to the right.  Now drop
blues into 5-2 and reds into 3-4.  You've written your third stripe,
rotating by two cups (disks) again.

This is pretty much how RAID6 works.  Each time we wrote we dropped 8
m&m's into 8 cups, 6 blue (data chunks) and 2 red (parity chunks).
Every RAID stripe you write will be constructed of 6 blues and 2 reds.
XFS, or EXT4, or any filesystem, can only drop blues into the first 6
cups of a stripe.  The RAID adds the two reds to every stripe.

Maybe now you understand why sw=6 for an 8 drive RAID6.  And now maybe
you understand what "distributed parity" actually means--every stripe is
shifted, not just the parity chunks.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: swidth in RAID

[aurfalien]

On Jun 30, 2013, at 2:42 PM, Stan Hoeppner wrote:

> On 6/30/2013 1:43 PM, aurfalien wrote:
> 
>> I understand swidth should = #data disks.
> 
> No.  "swidth" is a byte value specifying the number of 512 byte blocks
> in the data stripe.
> 
> "sw" is #data disks.
> 
>> And the docs say for RAID 6 of 8 disks, that means 6.
>> 
>> But parity is distributed and you actually have 8 disks/spindles working for you and a bit of parity on each.
>> 
>> So shouldn't swidth equal disks in raid when its concerning distributed parity raid?
> 
> No.  Lets try visual aids.
> 
> Set 8 coffee cups (disk drives) on a table.  Grab a bag of m&m's.
> Separate 24 blues (data) and 8 reds (parity).

But are the cups 8oz, 16oz or smaller/larger?

Ceramic, plastic, glass, etc...?

Actually I really enjoyed the visual aid, many many thanks.

- aurf

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: swidth in RAID

[Dave Chinner]

On Sun, Jun 30, 2013 at 04:42:06PM -0500, Stan Hoeppner wrote:
> On 6/30/2013 1:43 PM, aurfalien wrote:
> 
> > I understand swidth should = #data disks.
> 
> No.  "swidth" is a byte value specifying the number of 512 byte blocks
> in the data stripe.
> 
> "sw" is #data disks.
> 
> > And the docs say for RAID 6 of 8 disks, that means 6.
> > 
> > But parity is distributed and you actually have 8 disks/spindles working for you and a bit of parity on each.
> > 
> > So shouldn't swidth equal disks in raid when its concerning distributed parity raid?
> 
> No.  Lets try visual aids.
> 
> Set 8 coffee cups (disk drives) on a table.  Grab a bag of m&m's.
> Separate 24 blues (data) and 8 reds (parity).
> 
> Drop a blue m&m in cups 1-6 and a red into 7-8.  You just wrote one RAID
> stripe.  Now drop a blue into cups 3-8 and a red in 1-2.  Your second
> write, this time rotating two cups (drives) to the right.  Now drop
> blues into 5-2 and reds into 3-4.  You've written your third stripe,
> rotating by two cups (disks) again.
> 
> This is pretty much how RAID6 works.  Each time we wrote we dropped 8
> m&m's into 8 cups, 6 blue (data chunks) and 2 red (parity chunks).
> Every RAID stripe you write will be constructed of 6 blues and 2 reds.

Right, that's how they are constructed, but not all RAID distributes
parity across different disks in the array. Some are symmetric, some
are asymmetric, some rotate right, some rotate left, and some use
statistical algorithms to give an overall distribution without being
able to predict where a specific parity block might lie within a
stripe...

And at the other end of the scale, isochronous RAID arrays tend to
have dedicated parity disks so that data read and write behaviour is
deterministic and therefore predictable from a high level....

So, assuming that a RAID5/6 device has a specific data layout (be it
distributed or fixed) at the filesystem level is just a bad idea. We
simply don't know. Even if we did, the only thing we can optimise is
the thing that is common between all RAID5/6 devices - writing full
stripe widths is the most optimal method of writing to them....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: swidth in RAID

[aurfalien]

On Jun 30, 2013, at 6:38 PM, Dave Chinner wrote:

> On Sun, Jun 30, 2013 at 04:42:06PM -0500, Stan Hoeppner wrote:
>> On 6/30/2013 1:43 PM, aurfalien wrote:
>> 
>>> I understand swidth should = #data disks.
>> 
>> No.  "swidth" is a byte value specifying the number of 512 byte blocks
>> in the data stripe.
>> 
>> "sw" is #data disks.
>> 
>>> And the docs say for RAID 6 of 8 disks, that means 6.
>>> 
>>> But parity is distributed and you actually have 8 disks/spindles working for you and a bit of parity on each.
>>> 
>>> So shouldn't swidth equal disks in raid when its concerning distributed parity raid?
>> 
>> No.  Lets try visual aids.
>> 
>> Set 8 coffee cups (disk drives) on a table.  Grab a bag of m&m's.
>> Separate 24 blues (data) and 8 reds (parity).
>> 
>> Drop a blue m&m in cups 1-6 and a red into 7-8.  You just wrote one RAID
>> stripe.  Now drop a blue into cups 3-8 and a red in 1-2.  Your second
>> write, this time rotating two cups (drives) to the right.  Now drop
>> blues into 5-2 and reds into 3-4.  You've written your third stripe,
>> rotating by two cups (disks) again.
>> 
>> This is pretty much how RAID6 works.  Each time we wrote we dropped 8
>> m&m's into 8 cups, 6 blue (data chunks) and 2 red (parity chunks).
>> Every RAID stripe you write will be constructed of 6 blues and 2 reds.
> 
> Right, that's how they are constructed, but not all RAID distributes
> parity across different disks in the array. Some are symmetric, some
> are asymmetric, some rotate right, some rotate left, and some use
> statistical algorithms to give an overall distribution without being
> able to predict where a specific parity block might lie within a
> stripe...
> 
> And at the other end of the scale, isochronous RAID arrays tend to
> have dedicated parity disks so that data read and write behaviour is
> deterministic and therefore predictable from a high level....
> 
> So, assuming that a RAID5/6 device has a specific data layout (be it
> distributed or fixed) at the filesystem level is just a bad idea. We
> simply don't know. Even if we did, the only thing we can optimise is
> the thing that is common between all RAID5/6 devices - writing full
> stripe widths is the most optimal method of writing to them....

Am I interpreting this to say;

16 disks is sw=16 regardless of parity?

As the thing common is number of disks.   Or 1 parity as the least common denom which would mean sw=15?

Peter brought this up;

The main goal is trying to the reduce the probability of
read-modify-write.

Which is a way for me to think it as  "don't over subscribe".

- aurf

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: swidth in RAID

[Dave Chinner]

On Sun, Jun 30, 2013 at 06:54:31PM -0700, aurfalien wrote:
> 
> On Jun 30, 2013, at 6:38 PM, Dave Chinner wrote:
> 
> > On Sun, Jun 30, 2013 at 04:42:06PM -0500, Stan Hoeppner wrote:
> >> On 6/30/2013 1:43 PM, aurfalien wrote:
> >> 
> >>> I understand swidth should = #data disks.
> >> 
> >> No.  "swidth" is a byte value specifying the number of 512 byte blocks
> >> in the data stripe.
> >> 
> >> "sw" is #data disks.
> >> 
> >>> And the docs say for RAID 6 of 8 disks, that means 6.
> >>> 
> >>> But parity is distributed and you actually have 8 disks/spindles working for you and a bit of parity on each.
> >>> 
> >>> So shouldn't swidth equal disks in raid when its concerning distributed parity raid?
> >> 
> >> No.  Lets try visual aids.
> >> 
> >> Set 8 coffee cups (disk drives) on a table.  Grab a bag of m&m's.
> >> Separate 24 blues (data) and 8 reds (parity).
> >> 
> >> Drop a blue m&m in cups 1-6 and a red into 7-8.  You just wrote one RAID
> >> stripe.  Now drop a blue into cups 3-8 and a red in 1-2.  Your second
> >> write, this time rotating two cups (drives) to the right.  Now drop
> >> blues into 5-2 and reds into 3-4.  You've written your third stripe,
> >> rotating by two cups (disks) again.
> >> 
> >> This is pretty much how RAID6 works.  Each time we wrote we dropped 8
> >> m&m's into 8 cups, 6 blue (data chunks) and 2 red (parity chunks).
> >> Every RAID stripe you write will be constructed of 6 blues and 2 reds.
> > 
> > Right, that's how they are constructed, but not all RAID distributes
> > parity across different disks in the array. Some are symmetric, some
> > are asymmetric, some rotate right, some rotate left, and some use
> > statistical algorithms to give an overall distribution without being
> > able to predict where a specific parity block might lie within a
> > stripe...
> > 
> > And at the other end of the scale, isochronous RAID arrays tend to
> > have dedicated parity disks so that data read and write behaviour is
> > deterministic and therefore predictable from a high level....
> > 
> > So, assuming that a RAID5/6 device has a specific data layout (be it
> > distributed or fixed) at the filesystem level is just a bad idea. We
> > simply don't know. Even if we did, the only thing we can optimise is
> > the thing that is common between all RAID5/6 devices - writing full
> > stripe widths is the most optimal method of writing to them....
> 
> Am I interpreting this to say;
> 
> 16 disks is sw=16 regardless of parity?

No. I'm just saying that parity layout is irrelevant to the
filesystem and that all we care about is sw does not include parity
disks.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: swidth in RAID

[Stan Hoeppner]

On 6/30/2013 9:09 PM, Dave Chinner wrote:
> On Sun, Jun 30, 2013 at 06:54:31PM -0700, aurfalien wrote:
>>
>> On Jun 30, 2013, at 6:38 PM, Dave Chinner wrote:
>>
>>> On Sun, Jun 30, 2013 at 04:42:06PM -0500, Stan Hoeppner wrote:
>>>> On 6/30/2013 1:43 PM, aurfalien wrote:
>>>>
>>>>> I understand swidth should = #data disks.
>>>>
>>>> No.  "swidth" is a byte value specifying the number of 512 byte blocks
>>>> in the data stripe.
>>>>
>>>> "sw" is #data disks.
>>>>
>>>>> And the docs say for RAID 6 of 8 disks, that means 6.
>>>>>
>>>>> But parity is distributed and you actually have 8 disks/spindles working for you and a bit of parity on each.
>>>>>
>>>>> So shouldn't swidth equal disks in raid when its concerning distributed parity raid?
>>>>
>>>> No.  Lets try visual aids.
>>>>
>>>> Set 8 coffee cups (disk drives) on a table.  Grab a bag of m&m's.
>>>> Separate 24 blues (data) and 8 reds (parity).
>>>>
>>>> Drop a blue m&m in cups 1-6 and a red into 7-8.  You just wrote one RAID
>>>> stripe.  Now drop a blue into cups 3-8 and a red in 1-2.  Your second
>>>> write, this time rotating two cups (drives) to the right.  Now drop
>>>> blues into 5-2 and reds into 3-4.  You've written your third stripe,
>>>> rotating by two cups (disks) again.
>>>>
>>>> This is pretty much how RAID6 works.  Each time we wrote we dropped 8
>>>> m&m's into 8 cups, 6 blue (data chunks) and 2 red (parity chunks).
>>>> Every RAID stripe you write will be constructed of 6 blues and 2 reds.
>>>
>>> Right, that's how they are constructed, but not all RAID distributes
>>> parity across different disks in the array. Some are symmetric, some
>>> are asymmetric, some rotate right, some rotate left, and some use
>>> statistical algorithms to give an overall distribution without being
>>> able to predict where a specific parity block might lie within a
>>> stripe...
>>>
>>> And at the other end of the scale, isochronous RAID arrays tend to
>>> have dedicated parity disks so that data read and write behaviour is
>>> deterministic and therefore predictable from a high level....
>>>
>>> So, assuming that a RAID5/6 device has a specific data layout (be it
>>> distributed or fixed) at the filesystem level is just a bad idea. We
>>> simply don't know. Even if we did, the only thing we can optimise is
>>> the thing that is common between all RAID5/6 devices - writing full
>>> stripe widths is the most optimal method of writing to them....
>>
>> Am I interpreting this to say;
>>
>> 16 disks is sw=16 regardless of parity?
> 
> No. I'm just saying that parity layout is irrelevant to the
> filesystem and that all we care about is sw does not include parity
> disks.

So, here's the formula aurfalien, where #disks is the total number of
active disks (excluding spares) in the RAID array.  In the case of

RAID5	sw = (#disks - 1)
RAID6	sw = (#disks - 2)
RAID10  sw = (#disks / 2) [1]

[1] If using the Linux md/RAID10 driver with one of the non-standard
layouts such as n2 or f2, the formula may change.  This is beyond the
scope of this discussion.  Visit the linux-raid mailing list for further
details.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: swidth in RAID

[aurfalien]

On Jun 30, 2013, at 7:47 PM, Stan Hoeppner wrote:

> On 6/30/2013 9:09 PM, Dave Chinner wrote:
>> On Sun, Jun 30, 2013 at 06:54:31PM -0700, aurfalien wrote:
>>> 
>>> On Jun 30, 2013, at 6:38 PM, Dave Chinner wrote:
>>> 
>>>> On Sun, Jun 30, 2013 at 04:42:06PM -0500, Stan Hoeppner wrote:
>>>>> On 6/30/2013 1:43 PM, aurfalien wrote:
>>>>> 
>>>>>> I understand swidth should = #data disks.
>>>>> 
>>>>> No.  "swidth" is a byte value specifying the number of 512 byte blocks
>>>>> in the data stripe.
>>>>> 
>>>>> "sw" is #data disks.
>>>>> 
>>>>>> And the docs say for RAID 6 of 8 disks, that means 6.
>>>>>> 
>>>>>> But parity is distributed and you actually have 8 disks/spindles working for you and a bit of parity on each.
>>>>>> 
>>>>>> So shouldn't swidth equal disks in raid when its concerning distributed parity raid?
>>>>> 
>>>>> No.  Lets try visual aids.
>>>>> 
>>>>> Set 8 coffee cups (disk drives) on a table.  Grab a bag of m&m's.
>>>>> Separate 24 blues (data) and 8 reds (parity).
>>>>> 
>>>>> Drop a blue m&m in cups 1-6 and a red into 7-8.  You just wrote one RAID
>>>>> stripe.  Now drop a blue into cups 3-8 and a red in 1-2.  Your second
>>>>> write, this time rotating two cups (drives) to the right.  Now drop
>>>>> blues into 5-2 and reds into 3-4.  You've written your third stripe,
>>>>> rotating by two cups (disks) again.
>>>>> 
>>>>> This is pretty much how RAID6 works.  Each time we wrote we dropped 8
>>>>> m&m's into 8 cups, 6 blue (data chunks) and 2 red (parity chunks).
>>>>> Every RAID stripe you write will be constructed of 6 blues and 2 reds.
>>>> 
>>>> Right, that's how they are constructed, but not all RAID distributes
>>>> parity across different disks in the array. Some are symmetric, some
>>>> are asymmetric, some rotate right, some rotate left, and some use
>>>> statistical algorithms to give an overall distribution without being
>>>> able to predict where a specific parity block might lie within a
>>>> stripe...
>>>> 
>>>> And at the other end of the scale, isochronous RAID arrays tend to
>>>> have dedicated parity disks so that data read and write behaviour is
>>>> deterministic and therefore predictable from a high level....
>>>> 
>>>> So, assuming that a RAID5/6 device has a specific data layout (be it
>>>> distributed or fixed) at the filesystem level is just a bad idea. We
>>>> simply don't know. Even if we did, the only thing we can optimise is
>>>> the thing that is common between all RAID5/6 devices - writing full
>>>> stripe widths is the most optimal method of writing to them....
>>> 
>>> Am I interpreting this to say;
>>> 
>>> 16 disks is sw=16 regardless of parity?
>> 
>> No. I'm just saying that parity layout is irrelevant to the
>> filesystem and that all we care about is sw does not include parity
>> disks.
> 
> So, here's the formula aurfalien, where #disks is the total number of
> active disks (excluding spares) in the RAID array.  In the case of
> 
> RAID5	sw = (#disks - 1)
> RAID6	sw = (#disks - 2)
> RAID10  sw = (#disks / 2) [1]
> 
> [1] If using the Linux md/RAID10 driver with one of the non-standard
> layouts such as n2 or f2, the formula may change.  This is beyond the
> scope of this discussion.  Visit the linux-raid mailing list for further

I totally got your original post with the cup o M&Ms.

Just wanted his take on it is all.

And I'm on too many mailing lists as it is :)

- aurf
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: swidth in RAID

[Peter Grandi]

[ ... ]

>> RAID5	sw = (#disks - 1)
>> RAID6	sw = (#disks - 2)
>> RAID10       sw = (#disks / 2) [1]

What was probably all that needed saying for once is that
'swidth'/'sw' matter nearly only for avoiding read-modify-write,
and there is no reason to confuse the already confused by
mentioning here RAID10 (or RAID0) where read-modify-write won't
happen.

The somewhat secondary reason for which stripe width, or rather
something related to it, may matter even for non-parity RAID
sets is for filesystems that try to layout metadata tables so
that the metadata does not end up all on a subset of the disks
in the RAID set, which might occur if the metadata table
alignment is congruent with the "chunk" alignment.

That for example is likely to happen with 'ext[234]' filetrees,
and accordingly 'man mke2fs' rightly mentions for 'stripe-width'
(equivalent to 'swidth'/'sw') that is matters only for parity
RAID sets and because of read-modify-write:

  "This allows the block allocator to prevent read-modify-write
  of the parity in a RAID stripe if possible when the data is
  written."

and it is about 'stride' (the equivalent of 'su'/'sunit' in XFS)
that it says:

  "This mostly affects placement of filesystem metadata like
  bitmaps at mke2fs time to avoid placing them on a single disk,
  which can hurt performance.  It may also be used by the block
  allocator."

Uhm, I thougt that also affected placement of inode tables, but
I may be misremembering. Whether metadata alignment issues are
likley to happen with XFS, where metadata allocation is more
dynamic than for 'ext[234]', and whether it currently contains
code to deal with it, I don't remember.

Also, even assuming that 'sw' matters for RADI10 for reasons
other than parity updates that it does not do, the formula above
is simplistic:

>> [ ... ]
>> [1] If using the Linux md/RAID10 driver with one of the
>> non-standard layouts such as n2 or f2, the formula may
>> change. [ ... ]

Here the default is 'n' and the alternative layouts are 'o' and
'f', also with Linux MD there can be an odd number of members in
a RAID10 set. Not that matters as RAID10 (and some others) of
any shape does not have parity to update on write, so the
specific physical layouts of blocks is not relevant for RMW.

Anyhow I wrote a brief overall description of RMW here some time
ago:

  http://www.sabi.co.uk/blog/12-thr.html#120414

as RMW is an issue that matters in several cases other than
parity RAID.

Also because I think this is the third or fourth time that it
needed repeating in some mailing list that stripe width matters
almost only for RAID when there is parity, and thus almost
entirely not for RAID10.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: swidth in RAID

[Stan Hoeppner]

On 7/2/2013 4:48 PM, Peter Grandi wrote:

> What was probably all that needed saying for once is that

What was needed was a simple explanation demonstrating the answer the OP
was looking for Peter.  I provided that.

In your 10,000th attempt to generate self gratification by demonstrating
your superior knowledge (actually lack thereof) on this list, all you
could have possibly achieved here is confusing the OP even further.

I find it sad that you've decided to prey on the young, the
inexperienced, after realizing the educated began ignoring you long ago.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs