RAID10 Performance

RAID10 Performance

[Adam Goryachev]

Hi all,

I've got a system with the following config that I am trying to improve
performance on. Hopefully you can help guide me in the best direction
please.

1 x SSD (OS drive only)
3 x 2TB WDC WD2003FYYS-02W0B1

The three HDD's are configured in a single RAID10 (I configured as
RAID10 to easily support adding additional drives later, I realise/hope
it is currently equivalent to RAID1)
md0 : active raid10 sdb1[0] sdd1[2](S) sdc1[1]
      1953511936 blocks super 1.2 2 near-copies [2/2] [UU]

Which is then shared with DRBD to another identical system.
Then LVM is used to carve the redundant storage into virtual disks
Finally, iSCSI is used to export the virtual disks to the various
virtual machines running on other physical boxes.

When a single VM is accessing data, performance is more than acceptable
(max around 110M/s as reported by dd)

The two SAN machines have 1 Gb ethernet crossover between them, and 4 x
Gb bonded to the switch which connects to the physical machines running
the VM's (which have only a single Gb connection).

The issue is poor performance when more than one machine attempts to do
disk intensive activity at the same time (ie, when the anti virus scan
starts on all VM's at the same time, or during the backup window, etc).

During these times, performance can drop to 5M/s (reported by dd, or
calculated timings from windows VM etc). I'd like to:
a) improve overall performance when multiple VM's are r/w data to the drives
b) Hopefully set a minimum performance level for each VM (so one VM
can't starve the others).

I have adjusted some drbd related values, and significantly improved
performance there, not to say it is perfect yet).
I am currently using the deadline scheduler on the HDD's, but this
doesn't make much difference.
I have manually balanced IRQ's across the available CPU's (two bonded
ethernet on one, two bonded ethernet on second, SATA on third, and the
rest of the IRQ's on the fourth, it is a quad core CPU).

If I add the third (hot spare) disk into the RAID10 array, could I get
1.5x the total storage capacity, and improve performance by approx 30%?
If I add another two disks (on each server) could I extend the array to
2x total storage capacity, double performance, and still keep the hot spare?
If I add the third (hot spare) disk into the RAID10 array, could I get
1x the total storage capacity (ie, 3 disk RAID1) and improve read
performance?

Is there some other details I should provide, or knobs that I can tweak
to get better performance?

Additional data:
mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Mon Jun  4 23:31:20 2012
     Raid Level : raid10
     Array Size : 1953511936 (1863.01 GiB 2000.40 GB)
  Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
   Raid Devices : 2
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Fri Jul 27 00:07:21 2012
          State : active
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

         Layout : near=2
     Chunk Size : 512K

           Name : san2:0  (local to host san2)
           UUID : b402c62b:2ae7eca3:89422456:2cd7c6f3
         Events : 82

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1

       2       8       49        -      spare   /dev/sdd1

cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
    ns:295435862 nr:0 dw:298389532 dr:1582870916 al:10877089 bm:53752
lo:2 pe:0 ua:0 ap:1 ep:1 wo:b oos:0

Running debian stable 2.6.32-5-amd64

top - 00:10:57 up 20 days, 18:41,  1 user,  load average: 0.25, 0.50, 0.61
Tasks: 340 total,   1 running, 339 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.0%st
Mem:   7903292k total,  7702696k used,   200596k free,  5871076k buffers
Swap:  3939320k total,        0k used,  3939320k free,  1266756k cached

(Note, CPU load peaks up to 12 during heavy IO load periods)

Thank you for any advice or assistance you can provide.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: RAID10 Performance

[Stan Hoeppner]

On 7/26/2012 9:16 AM, Adam Goryachev wrote:

> I've got a system with the following config that I am trying to improve
> performance on. Hopefully you can help guide me in the best direction
> please.

> 3 x 2TB WDC WD2003FYYS-02W0B1
...
> The three HDD's are configured in a single RAID10
...
> Which is then shared with DRBD to another identical system.
> Then LVM is used to carve the redundant storage into virtual disks
> Finally, iSCSI is used to export the virtual disks to the various
> virtual machines running on other physical boxes.
> 
> When a single VM is accessing data, performance is more than acceptable
> (max around 110M/s as reported by dd)
> 
> The two SAN machines have 1 Gb ethernet crossover between them, and 4 x
> Gb bonded to the switch which connects to the physical machines running
> the VM's (which have only a single Gb connection).
> 
> The issue is poor performance when more than one machine attempts to do
> disk intensive activity at the same time (ie, when the anti virus scan
> starts on all VM's at the same time, or during the backup window, etc).
...

I'm really surprised you don't already know the answer, and that you
gave such a lengthy detailed description.

Your problem is very simple.  It is suffered by many people, who lack
basic understanding of rotating drive performance in relation to their
workloads.

You don't have enough seek bandwidth.  The drive heads simply can't move
fast enough to service all sector requests in a timely manner.  There is
no way to fix this by tweaking the operating system.  You need to
increase your seek rate.

1.  Recreate the arrays with 6 or 8 drives each, use a 64KB chunk
2.  Replace the 7.2k WD drives with 10k SATA, or 15k SAS drives
3.  Replace the drives with SSDs

Any of these 3 things will decrease latency per request.

-- 
Stan

Re: RAID10 Performance

[Phil Turmel]

Hi Adam,

On 07/26/2012 10:16 AM, Adam Goryachev wrote:
> Hi all,
> 
> I've got a system with the following config that I am trying to improve
> performance on. Hopefully you can help guide me in the best direction
> please.
> 
> 1 x SSD (OS drive only)
> 3 x 2TB WDC WD2003FYYS-02W0B1
> 
> The three HDD's are configured in a single RAID10 (I configured as
> RAID10 to easily support adding additional drives later, I realise/hope
> it is currently equivalent to RAID1)
> md0 : active raid10 sdb1[0] sdd1[2](S) sdc1[1]
>       1953511936 blocks super 1.2 2 near-copies [2/2] [UU]

In addition to Stan's comments, you should know that MD raid10 is not
reshapeable yet.  You cannot use --grow to increase that array's
capacity.

That feature is in the roadmap, so you can expect it eventually.

HTH,

Phil

Re: RAID10 Performance

[Adam Goryachev]

On 27/07/12 17:07, Stan Hoeppner wrote:
> On 7/26/2012 9:16 AM, Adam Goryachev wrote:
> 
>> I've got a system with the following config that I am trying to
>> improve performance on. Hopefully you can help guide me in the
>> best direction please.
> 
>> 3 x 2TB WDC WD2003FYYS-02W0B1
> ...
>> The three HDD's are configured in a single RAID10
> ...
>> Which is then shared with DRBD to another identical system. Then
>> LVM is used to carve the redundant storage into virtual disks 
>> Finally, iSCSI is used to export the virtual disks to the
>> various virtual machines running on other physical boxes.
>> 
>> When a single VM is accessing data, performance is more than
>> acceptable (max around 110M/s as reported by dd)
>> 
>> The two SAN machines have 1 Gb ethernet crossover between them,
>> and 4 x Gb bonded to the switch which connects to the physical
>> machines running the VM's (which have only a single Gb
>> connection).
>> 
>> The issue is poor performance when more than one machine attempts
>> to do disk intensive activity at the same time (ie, when the anti
>> virus scan starts on all VM's at the same time, or during the
>> backup window, etc).
> ...
> 
> I'm really surprised you don't already know the answer, and that
> you gave such a lengthy detailed description.

Ummm, yes, well I was hoping for some magic I didn't already know :)

> Your problem is very simple.  It is suffered by many people, who
> lack basic understanding of rotating drive performance in relation
> to their workloads.
> 
> You don't have enough seek bandwidth.  The drive heads simply can't
> move fast enough to service all sector requests in a timely manner.
> There is no way to fix this by tweaking the operating system.  You
> need to increase your seek rate.
> 
> 1.  Recreate the arrays with 6 or 8 drives each, use a 64KB chunk

Would you suggest these 6 - 8 drives in RAID10 or some other RAID
level? (IMHO, the best performance with reasonable protection is RAID10)

How do you get that many drives into a decent "server"? I'm using a
4RU rackmount server case, but it only has capacity for 5 x hot swap
3.5" drives (plus one internal drive).

> 2.  Replace the 7.2k WD drives with 10k SATA, or 15k SAS drives

Which drives would you suggest? The drives I have are already over
$350 each (AUD)...

> 3.  Replace the drives with SSDs

Yes, I'd love to do this.

> Any of these 3 things will decrease latency per request.

I have already advised adding an additional pair of drives, and
converting to SSD's.

Would adding another 2 identical drives and configuring in RAID10
really improve performance by double?

Would it be more than double (because much less seeking should give
better throughput, similar I expect to performance of two concurrent
reads is less than half of a single read)?

If using SSD's, what would you suggest to get 1TB usable space?
Would 4 x Intel 480GB SSD 520 Series (see link) in RAID10 be the best
solution? Would it make more sense to use 4 in RAID6 so that expansion
is easier in future (ie, add a 5th drive to add 480G usable storage)?

http://www.megaware.com.au/index.php?main_page=product_info&cPath=16_1579&products_id=133503

I'm just trying to get some real life experiences from others. This
system is pretty much the highest performing system I've built to date....

Thank you for your comments, I do appreciate them.

Regards,
Adam

PS, thanks for the reminder that RAID10 grow is not yet supported, I
may need to do some creative raid management to "grow" the array,
extended downtime is possible to get that done when needed...

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: RAID10 Performance

[Stan Hoeppner]

On 7/27/2012 8:02 AM, Adam Goryachev wrote:
> On 27/07/12 17:07, Stan Hoeppner wrote:

>> 1.  Recreate the arrays with 6 or 8 drives each, use a 64KB chunk
> 
> Would you suggest these 6 - 8 drives in RAID10 or some other RAID
> level? (IMHO, the best performance with reasonable protection is RAID10)

You're running many VMs.  That implies a mixed read/write workload.  You
can't beat RAID10 for this workload.

> How do you get that many drives into a decent "server"? I'm using a
> 4RU rackmount server case, but it only has capacity for 5 x hot swap
> 3.5" drives (plus one internal drive).

Given that's a 4U chassis, I'd bet you mean 5 x 5.25" bays with 3.5" hot
swap carriers.

In that case, get 8 of these:
http://www.newegg.com/Product/Product.aspx?Item=N82E16822148710

and replace two carriers with two of these:
http://www.newegg.com/Product/Product.aspx?Item=N82E16817994142

Providing the make/model of the case would be helpful.

>> 2.  Replace the 7.2k WD drives with 10k SATA, or 15k SAS drives
> 
> Which drives would you suggest? The drives I have are already over
> $350 each (AUD)...

See above.  Or you could get 6 of these:
http://www.newegg.com/Product/Product.aspx?Item=N82E16822236243

>> 3.  Replace the drives with SSDs
> 
> Yes, I'd love to do this.
> 
>> Any of these 3 things will decrease latency per request.
> 
> I have already advised adding an additional pair of drives, and
> converting to SSD's.
> 
> Would adding another 2 identical drives and configuring in RAID10
> really improve performance by double?

No because you'd have 5 drives and you have 3 now.  With 6 drives total,
yes, performance will be doubled.  And BTW, there is no such thing as an
odd number of drives RAID10.  That's actually more akin to RAID1E.  It
just happens that the md/RAID driver that provides it is the RAID10
driver.  You want an even number of drives. Use a standard RAID10
layout, not "near" or "far" layouts.

> Would it be more than double (because much less seeking should give
> better throughput, similar I expect to performance of two concurrent
> reads is less than half of a single read)?

It's not throughput you're after, but latency reduction.  More spindles
in the stripe, or faster spindles, is what you want for a high
concurrency workload, to reduce latency.  Reducing latency will increase
the responsiveness of the client machines, and, to some degree
throughput, because the array can now process more IO transactions per
unit time.

> If using SSD's, what would you suggest to get 1TB usable space?
> Would 4 x Intel 480GB SSD 520 Series (see link) in RAID10 be the best
> solution? Would it make more sense to use 4 in RAID6 so that expansion
> is easier in future (ie, add a 5th drive to add 480G usable storage)?

I actually wouldn't recommend SSDs for this server.  The technology is
still young, hasn't been proven yet for long term server use, and the
cost/GB is still very high, more than double that of rust.

I'd recommend the 10k RPM WD Raptor 1TB drives.  They're sold as 3.5"
drives but are actually 2.5" drives in a custom mounting frame, so you
can use them in a chassis with either size of hot swap cage.  They're
also very inexpensive given the performance plus capacity.

I'd also recommend using XFS if you aren't already.  And do NOT use the
metadata 1.2 default 512KB chunk size.  It is horrible for random write
workloads.  Use a 32KB chunk with your md/RAID10 with these fast drives,
and align XFS during mkfs.xfs using -d options su/sw.

> PS, thanks for the reminder that RAID10 grow is not yet supported, I
> may need to do some creative raid management to "grow" the array,
> extended downtime is possible to get that done when needed...

You can grow a RAID10 based array:  join 2 or more 4 drive RAID10s in a
--linear array.  Add more 4 drive RAID10s in the future by growing the
linear array.  Then grow the filesystem over the new space.

-- 
Stan

Re: RAID10 Performance

[Adam Goryachev]

On 28/07/12 04:29, Stan Hoeppner wrote:
> On 7/27/2012 8:02 AM, Adam Goryachev wrote:
>> On 27/07/12 17:07, Stan Hoeppner wrote:
> 
>>> 1.  Recreate the arrays with 6 or 8 drives each, use a 64KB 
>>> chunk
>> How do you get that many drives into a decent "server"? I'm
>> using a 4RU rackmount server case, but it only has capacity for 5
>> x hot swap 3.5" drives (plus one internal drive).
> 
> Given that's a 4U chassis, I'd bet you mean 5 x 5.25" bays with 
> 3.5" hot swap carriers.

I'm not exactly sure the number of external 5.25" bays at the moment,
but at least 3, possibly 4 or 5. Currently I have a extra "chassis"
that provides 5 x 3.5" hot swap bays using 3 x 5.25" bays. I can also
get a similar unit that will convert 2 x 5.25" to 3 x 3.5" hot swap
bays....

> In that case, get 8 of these: 
> http://www.newegg.com/Product/Product.aspx?Item=N82E16822148710

Are you suggesting these drives because:
a) They perform better than the WD drives?
b) They are cheaper than the WD drives
c) give more spindles per TB
d) The physical size
e) other?

Just trying to clarify the choices, as far as I can find, the avg seek
times are almost identical, but for reason b and c, I could see the
advantage.

> and replace two carriers with two of these: 
> http://www.newegg.com/Product/Product.aspx?Item=N82E16817994142

Thank you for your suggestions.

> Providing the make/model of the case would be helpful.

Don't have it handy right now, but I think with the above suggestions
I've got enough :)

>>> 2.  Replace the 7.2k WD drives with 10k SATA, or 15k SAS 
>>> drives
>> 
>> Which drives would you suggest? The drives I have are already 
>> over $350 each (AUD)...
> 
> See above.  Or you could get 6 of these: 
> http://www.newegg.com/Product/Product.aspx?Item=N82E16822236243

Would 6 of these perform better than 8 of the above seagates at 7200rpm?

>> Would adding another 2 identical drives and configuring in RAID10
>> really improve performance by double?
> 
> No because you'd have 5 drives and you have 3 now.

Sorry, I wasn't clear. I currently am using a 2 drive RAID10 (which is
the same as a RAID1) with a hot spare. The third drive is not active.

> With 6 drives total, yes, performance will be doubled.  And BTW, 
> there is no such thing as an odd number of drives RAID10.  That's 
> actually more akin to RAID1E.  It just happens that the md/RAID 
> driver that provides it is the RAID10 driver.  You want an even 
> number of drives. Use a standard RAID10 layout, not "near" or
> "far" layouts.

Certainly :)

>> Would it be more than double (because much less seeking should 
>> give better throughput, similar I expect to performance of two 
>> concurrent reads is less than half of a single read)?
> 
> It's not throughput you're after, but latency reduction.  More 
> spindles in the stripe, or faster spindles, is what you want for a 
> high concurrency workload, to reduce latency.  Reducing latency 
> will increase the responsiveness of the client machines, and, to 
> some degree throughput, because the array can now process more IO 
> transactions per unit time.

The specific workload that performance is being measured on is
actually large file read + concurrent large file write + concurrent
small random read/write. ie, in better english:
1) Normal operations (small random read/write, low load)
2) Performance testing - copying a large file with source and
destination on the same location.

In real world application, number 2 is replaced by a once weekly
"maintenance procedure" that essentially is a backup (copy large file
from/to same drive).

In actual fact, normal performance currently is fine, apart from this
weekly maintenance task (blackbox, I'm not allowed to know more).

The main concern is that once we add a SQL server, domain controller,
and 2 XP VM's, it will further increase the stress on the system.

>> If using SSD's, what would you suggest to get 1TB usable space? 
>> Would 4 x Intel 480GB SSD 520 Series (see link) in RAID10 be the 
>> best solution? Would it make more sense to use 4 in RAID6 so
>> that expansion is easier in future (ie, add a 5th drive to add
>> 480G usable storage)?
> 
> I actually wouldn't recommend SSDs for this server.  The
> technology is still young, hasn't been proven yet for long term
> server use, and the cost/GB is still very high, more than double
> that of rust.
> 
> I'd recommend the 10k RPM WD Raptor 1TB drives.  They're sold as 
> 3.5" drives but are actually 2.5" drives in a custom mounting 
> frame, so you can use them in a chassis with either size of hot 
> swap cage.  They're also very inexpensive given the performance 
> plus capacity.

Would you suggest that these drives are reliable enough to support
this type of usage? We are currently using enterprise grade drives...

> I'd also recommend using XFS if you aren't already.  And do NOT
> use the metadata 1.2 default 512KB chunk size.  It is horrible for 
> random write workloads.  Use a 32KB chunk with your md/RAID10 with 
> these fast drives, and align XFS during mkfs.xfs using -d options 
> su/sw.

Not using XFS at all, it's just plain raw disk space from MD, LVM2,
and exported via iSCSI, the client VM will format (NTFS) and use it.

>> PS, thanks for the reminder that RAID10 grow is not yet 
>> supported, I may need to do some creative raid management to 
>> "grow" the array, extended downtime is possible to get that done 
>> when needed...
> 
> You can grow a RAID10 based array:  join 2 or more 4 drive RAID10s 
> in a --linear array.  Add more 4 drive RAID10s in the future by 
> growing the linear array.  Then grow the filesystem over the new 
> space.

Does a 8 drive RAID10 look like:
A A B B C C D D
...
W W X X Y Y Z Z

OR

A A B B W W X X
...
C C D D Y Y Z Z

In other words, does RAID10 with 8 drives write 4 x as fast as a
single drive (large continuous write) by splitting it into 4 stripes
and writing each stripe to a pair of drives.

Just in case I'm being silly, could I create a 8 drive RAID10 array
using the drives you suggested above, giving 4TB usable space, move
the existing 3 drives to the "standby" server, giving it 6 x 2TB
drives in RAID10 maybe 2 x hot spare, and 4 usable for 4TB total
usable space?

Long term, the "standby" SAN could be replaced with the same 8 x 1TB
drives, and move the 6 x 2TB drives into the disk based backup server
(not san). This would avoid wasting the drives.

Thanks again for your comments and suggestions on parts.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: RAID10 Performance

[Stan Hoeppner]

On 7/28/2012 1:36 AM, Adam Goryachev wrote:
> On 28/07/12 04:29, Stan Hoeppner wrote:

>> In that case, get 8 of these: 
>> http://www.newegg.com/Product/Product.aspx?Item=N82E16822148710
> 
> Are you suggesting these drives because:
> a) They perform better than the WD drives?
> b) They are cheaper than the WD drives
> c) give more spindles per TB
> d) The physical size
> e) other?

c&d, d facilitates c

> Just trying to clarify the choices, as far as I can find, the avg seek
> times are almost identical, but for reason b and c, I could see the
> advantage.

They're not cheaper per GB, but yes, you can get ~twice the spindles in
the same space, which is the key to performance here if sticking with
7.2k spindles.

But I think you should go with the 10K rpm Raptors.  Same capacity but
with a 40% increase in spindle speed for only 30% more cost, at Newegg
prices anyway, but I don't think Newegg ships to Australia.  If money
were of no concern, which is rarely the case, I'd recommend 15K drives.
 But they're just so disproportionately expensive compared to 10K drives
given the capacities offered.

>> See above.  Or you could get 6 of these: 
>> http://www.newegg.com/Product/Product.aspx?Item=N82E16822236243
> 
> Would 6 of these perform better than 8 of the above seagates at 7200rpm?

Generally speaking, the performance should be similar.  Choosing between
7.2/10/15k drives is a tradeoff between spindle speed, count, and
capacity.  These numbers are generally accepted estimates of spindle
performance across all drives from all vendors.  These are "peak" random
head seek rates.

7.2k = 150 IOPS = 8*150= 1200 IOPS
10k  = 225 IOPS = 6*225= 1350 IOPS
15k  = 300 IOPS = 4*300= 1200 IOPS

Four 15k drives will give the same IOPS as 8*7.2k drives for slightly
less total money, but you'll get only about 1/8th the capacity with
600GB 15K drives and 2TB 7.2K drives, 2.4TB vs 16TB.  But you can fit
almost twice as many drives in the same chassis space.  So if
performance is more critical than budget or space and you max your
chassis, your total IOPS will be 4800 vs 1200 w/16 vs 8 drives, in the
same rack space.  Which is exactly why you see many rows of racks in
many enterprise datacenters housing 2.5" drives--performance is more
critical than cost, and they have plenty of racks--and money--to achieve
the needed capacity.

You mentioned adding more capacity in the near future.  It is always
better to buy as much up front as possible and expand as little as
possible as this prevents disruption to your production environment.

If cost isn't an overriding concern, my recommendation would be to add 8
of the 10k 1TB Raptor drives and use them for your iSCSI LUN exports,
and redeploy the RE4 drives.

The performance gain with either 6 or 8 of the Raptors will be substantial.

>>> Would adding another 2 identical drives and configuring in RAID10
>>> really improve performance by double?
>>
>> No because you'd have 5 drives and you have 3 now.
> 
> Sorry, I wasn't clear. I currently am using a 2 drive RAID10 (which is
> the same as a RAID1) with a hot spare. The third drive is not active.

Ah, ok.  Yeah, you could add 2 drives and double your IOPS, but you'd
have to scrap and rebuild the array, dumping and restoring any data, as
md/RAID10 can't currently be expanded/grown.

> The specific workload that performance is being measured on is
> actually large file read + concurrent large file write + concurrent
> small random read/write. ie, in better english:
> 1) Normal operations (small random read/write, low load)
> 2) Performance testing - copying a large file with source and
> destination on the same location.

The second turns a big streaming workload into a big random workload due
to head thrashing between two platter regions.  The 10K drives will
definitely help here, especially with more of them.  At least 6 or 8.

And don't use the default 512KB chunk size of metadata 1.2.  512KB per
chunk is insane.  With your Win server VM workload, where no server does
much writing of large files or at a sustained rate, usually only small
files, you should be using a small chunk size, something like 32KB,
maybe even 16KB.  If you use a large chunk size you'll rarely be able to
fill a full stripe write, and you'll end up with IO hot spots on
individual drives, decreasing performance.

> In real world application, number 2 is replaced by a once weekly
> "maintenance procedure" that essentially is a backup (copy large file
> from/to same drive).

Are you describing snapshotting a LUN here?

> In actual fact, normal performance currently is fine, apart from this
> weekly maintenance task (blackbox, I'm not allowed to know more).

Because the Windows VMs don't do much IO, makes sense.

> The main concern is that once we add a SQL server, domain controller,
> and 2 XP VM's, it will further increase the stress on the system.

That's simply untenable with the 2 spindles you currently have.

Assuming the SQL server actually does a little work, that VM alone may
likely generate more daily IO than all others combined.

This, overall, is a typical random IOPS server workload, at least from
the perspective of the RAID subsystem in this machine.  All of the large
file operations you mention are mixed with either other large file
operations or small file ops, making the overall IO pattern a mixed
random IOPS workload.  Again, you need faster spindles, more of them,
preferably both.  Adding 8*10k 1TB drives will make a world of difference.

>> I'd recommend the 10k RPM WD Raptor 1TB drives.  They're sold as 
>> 3.5" drives but are actually 2.5" drives in a custom mounting 
>> frame, so you can use them in a chassis with either size of hot 
>> swap cage.  They're also very inexpensive given the performance 
>> plus capacity.
> 
> Would you suggest that these drives are reliable enough to support
> this type of usage? We are currently using enterprise grade drives...

The WD VelociRaptors are enterprise grade drives, just as the RE4 and XE
series.  They have the same TLER/ERC support for hardware RAID use and
the same 5 year enterprise warranty.  They're tested and certified by
LSI for use on every LSI RAID controller, just as the other two.  No
other WD drives are certified but these 3 lines.  I assume you're
familiar with LSI, the world's leading and largest producer of high
performance branded and OEM enterprise RAID HBAs, producers of Dell and
IBM's OEM RAID cards, the PERC and ServeRAID lines?

I wouldn't have mentioned them if I didn't believe they are suitable. ;)

> Not using XFS at all, it's just plain raw disk space from MD, LVM2,
> and exported via iSCSI, the client VM will format (NTFS) and use it.

You mentioned that in your first post and I simply missed/forgot it.
Sorry bout that.

>> You can grow a RAID10 based array:  join 2 or more 4 drive RAID10s 
>> in a --linear array.  Add more 4 drive RAID10s in the future by 
>> growing the linear array.  Then grow the filesystem over the new 
>> space.

Again, missed that you're exporting raw device space.

> Does a 8 drive RAID10 look like:
> A A B B C C D D
> ...
> W W X X Y Y Z Z
> 
> OR
> 
> A A B B W W X X
> ...
> C C D D Y Y Z Z

The actual pattern is a bit irrelevant.

> In other words, does RAID10 with 8 drives write 4 x as fast as a
> single drive (large continuous write) by splitting it into 4 stripes
> and writing each stripe to a pair of drives.

In a 'perfect' scenario such as a massive large file write with an
appropriate chunk/stripe size, overall array streaming write performance
should be close to 4x that of a single drive, assuming there are no
hardware bottlenecks slowing down the duplicate writes to the mirror
drives, and no CPU saturation.

An 8 drive RAID 10 has 4 data spindles and 4 redundancy spindles--in
essence a stripe over 4 RAID1 pairs.  That's the standard RAID10 layout.
 The non standard layouts unique to md will yield considerably different
layout and performance characteristics.  They can also decrease the
number of concurrent drive failures the array can survive vs standard
RAID10.  For these reasons and others I don't care for these alternate
layouts.

> Just in case I'm being silly, could I create a 8 drive RAID10 array
> using the drives you suggested above, giving 4TB usable space, move
> the existing 3 drives to the "standby" server, giving it 6 x 2TB
> drives in RAID10 maybe 2 x hot spare, and 4 usable for 4TB total
> usable space?

In this scenario the standby server will have 1/3rd the disk IOPS of the
8 drive primary machine, assuming you use the 10K Raptor drives.  Most
people are using single GbE for DRBD, limited to ~100MB/s, so the
difference in array write speed shouldn't be a problem for DRBD.  Just
make sure you have your block device mirroring setup right and you
should be fine.

And of course you'll have ~1/3rd the IOPS and throughput should you have
to deploy the standby in production.

Many people run a DRBD standby server of lesser performance than their
primary, treating it as a hedge against primary failure, and assuming
they'll never have to use it.  But it's there just in case.  Thus they
don't put as much money or capability in it.  I.e. you'd have lots of
company if you did this.

> Long term, the "standby" SAN could be replaced with the same 8 x 1TB
> drives, and move the 6 x 2TB drives into the disk based backup server
> (not san). This would avoid wasting the drives.

That sounds like a perfectly good strategy to me.

> Thanks again for your comments and suggestions on parts.

You're welcome Adam.

Did you happen to notice the domain in my email address? ;)  If you need
hardware information/advice, on anything from channel
CPUs/mobos/drives/RAID/NICs/etc to 2560 CPU SGI supercomputers and 1200+
drive FC SAN storage arrays, FC switch fabrics, and anything in between,
I can usually provide the info you seek.

-- 
Stan

Re: RAID10 Performance

[Adam Goryachev]

Just some followup questions, hopefully this isn't too off-topic for 
this list, if it is please let me know.

On 07/29/2012 01:33 AM, Stan Hoeppner wrote:
> On 7/28/2012 1:36 AM, Adam Goryachev wrote:
>> On 28/07/12 04:29, Stan Hoeppner wrote:
> But I think you should go with the 10K rpm Raptors.  Same capacity but
> with a 40% increase in spindle speed for only 30% more cost, at Newegg
> prices anyway, but I don't think Newegg ships to Australia.  If money
> were of no concern, which is rarely the case, I'd recommend 15K drives.
>   But they're just so disproportionately expensive compared to 10K drives
> given the capacities offered.
>
> If cost isn't an overriding concern, my recommendation would be to add 8
> of the 10k 1TB Raptor drives and use them for your iSCSI LUN exports,
> and redeploy the RE4 drives.
>
> The performance gain with either 6 or 8 of the Raptors will be substantial.

OK, with the given budget, we currently have a couple of options:
1) Replace the primary SAN (which currently has 2 x 2TB WD Caviar Black 
RE4 drives in RAID1 + a hot spare) with 5 x 1TB Raptors you suggested 
above (4 x 1TB in RAID10 + 1 hot spare).

2) Replace the primary SAN with 3 x 480GB SSD drives in linear + one of 
the existing 2TB drives combined as RAID1 with the 2TB drive in write 
only mode. This reduces overall capacity, but it does provide enough 
capacity for at least 6 to 12 months. If needed, one additional SSD will 
provide almost 2TB across the entire san.

Further expansion becomes expensive, but this enterprise doesn't have a 
lot of data growth (over the past 10 years), so I don't expect it to be 
significant, also given the rate of increasing SSD storage, and 
decreasing cost. Long term it would be ideal to make this system use 8 x 
480G in RAID10, and eventually another 8 on the secondary SAN.

I'm aware that a single SSD failure will reduce performance back to 
current levels.

> And don't use the default 512KB chunk size of metadata 1.2.  512KB per
> chunk is insane.  With your Win server VM workload, where no server does
> much writing of large files or at a sustained rate, usually only small
> files, you should be using a small chunk size, something like 32KB,
> maybe even 16KB.  If you use a large chunk size you'll rarely be able to
> fill a full stripe write, and you'll end up with IO hot spots on
> individual drives, decreasing performance.

I'm assuming that this can't be changed?

Currently, I have:
md RAID
DRBD
LVM

Could I simply create a new MD array with the smaller chunk size, tell 
DRBD to sync from the remote to this new array, and then do the same for 
the other SAN?

Would this explain why one drive has significantly more activity now? I 
don't think so, since it is really just a 2 disk RAID1, so both drives 
should be doing the same writes, and both drives should be servicing the 
read requests. This doesn't seem to be happening, during very high read 
IO this morning (multiple VM's running a full antivirus scan 
simultaneously), one drive activity light was "solid" and the second was 
"flashing slowly".

Added to this, I've just noticed that the array is currently doing a 
"check", would that explain the drive activity and also reduced performance?

> And of course you'll have ~1/3rd the IOPS and throughput should you have
> to deploy the standby in production.
>
> Many people run a DRBD standby server of lesser performance than their
> primary, treating it as a hedge against primary failure, and assuming
> they'll never have to use it.  But it's there just in case.  Thus they
> don't put as much money or capability in it.  I.e. you'd have lots of
> company if you did this.

Understood, and in the case of primary SAN failure, at least work can 
still be completed, even if it is at reduced performance. Replacement 
drives can be done within one working day, so we are further limiting 
that reduced performance to one working day. This would be a perfect 
risk strategy for us, though again, long term we will look at upgrading 
the secondary san to be more similar to the primary.

> Did you happen to notice the domain in my email address? ;)  If you need
> hardware information/advice, on anything from channel
> CPUs/mobos/drives/RAID/NICs/etc to 2560 CPU SGI supercomputers and 1200+
> drive FC SAN storage arrays, FC switch fabrics, and anything in between,
> I can usually provide the info you seek.

My direct emails to you bounced due to my mail server "losing" it's 
reverse DNS. That is a local matter, delayed while "proving" we have 
ownership of the IP space to APNIC.

Thanks again for your advise.

Regards,
Adam

Re: RAID10 Performance

[Stan Hoeppner]

On 8/7/2012 10:49 PM, Adam Goryachev wrote:
> Just some followup questions, hopefully this isn't too off-topic for
> this list, if it is please let me know.

> OK, with the given budget, we currently have a couple of options:
> 1) Replace the primary SAN (which currently has 2 x 2TB WD Caviar Black
> RE4 drives in RAID1 + a hot spare) with 5 x 1TB Raptors you suggested
> above (4 x 1TB in RAID10 + 1 hot spare).

Wow, that is a tight budget.  You'll be increasing total IOPS by about
3x with 4 10k drives.  Probably not optimal for your workload but it's a
good start.

Also, don't use a hot spare--just wastes ports, cages, dollars.  It's
not needed with RAID10.  Keep a spare in a locked cabinet and hot swap
the drives upon failure notification.  RAID10 has a very large "double
drive failure kills array" window, like many months to years.

BTW, Caviar Black != RE4  They're different products.

> 2) Replace the primary SAN with 3 x 480GB SSD drives in linear + one of
> the existing 2TB drives combined as RAID1 with the 2TB drive in write
> only mode. This reduces overall capacity, but it does provide enough
> capacity for at least 6 to 12 months. If needed, one additional SSD will
> provide almost 2TB across the entire san.

This is simply insane, frankly.  Don't attempt this, even if md does
support a write only mirror partner.

> Further expansion becomes expensive, but this enterprise doesn't have a
> lot of data growth (over the past 10 years), so I don't expect it to be
> significant, also given the rate of increasing SSD storage, and
> decreasing cost. Long term it would be ideal to make this system use 8 x
> 480G in RAID10, and eventually another 8 on the secondary SAN.

Wait until *enterprise* SSD is fully mature and less expensive.  Stick
with mechanical storage for now as your budget doesn't support SSD.
If/when you go SSD, go *all* SSD, not this asymmetric Frankenstein
stuff, which will only cause you problems.

> I'm aware that a single SSD failure will reduce performance back to
> current levels.

Then why would you ever consider this?  This thread is about increasing
performance.  Why would you build a system that will instantly decrease
IOPS by a factor of 1000 upon device failure?  That's insane.

>> And don't use the default 512KB chunk size of metadata 1.2.  512KB per
>> chunk is insane.  With your Win server VM workload, where no server does
>> much writing of large files or at a sustained rate, usually only small
>> files, you should be using a small chunk size, something like 32KB,
>> maybe even 16KB.  If you use a large chunk size you'll rarely be able to
>> fill a full stripe write, and you'll end up with IO hot spots on
>> individual drives, decreasing performance.
> 
> I'm assuming that this can't be changed?

You assume what can't be changed?  Define what is changing.  It's a
simple command line switch to change the chunk size from the default.
See man mdadm.

> Currently, I have:
> md RAID
> DRBD
> LVM
> 
> Could I simply create a new MD array with the smaller chunk size, tell

So you're asking how to migrate from the current disks to the new disks?
 Yes you obviously must create a new RAID10 array.  Do you have enough
SAS/SATA ports to have all (2+4) disks running?  If so this should be
straightforward but will require some downtime.

> DRBD to sync from the remote to this new array, and then do the same for
> the other SAN?

Don't do this over ethernet.  What I would do is simply shut down all
daemons that may write to the current array, make it "static".  Shut
down DRBD on both hosts.  Use dd or a partition tool to copy everything
from the 2TB md/RAID1 mirror to the new 4 drive array.  Change your
mounts etc to the new RAID10 device.  Down the 2TB mirror array.
Confirm the new array is working.  Delete the current DRBD configuration
on both hosts and create a new one.  Start DRBD on both hosts and as its
syncing up restart services.

> Would this explain why one drive has significantly more activity now? I
> don't think so, since it is really just a 2 disk RAID1, so both drives
> should be doing the same writes, and both drives should be servicing the
> read requests. This doesn't seem to be happening, during very high read
> IO this morning (multiple VM's running a full antivirus scan
> simultaneously), one drive activity light was "solid" and the second was
> "flashing slowly".

md/RAID1 doesn't guarantee symmetric read IO across members of the pair.
 RAID1 isn't for high performance.  It's for cheap redundancy.  Even
RAID0 can exhibit this behavior if you have a large chunk size and lots
of small files.  They tend to align to the first drive in the stripe
because they fit entirely within the chunk.  This is why a default 512KB
chunk is insane for most workloads, especially mail servers.

> Added to this, I've just noticed that the array is currently doing a
> "check", would that explain the drive activity and also reduced
> performance?

Of course.

> My direct emails to you bounced due to my mail server "losing" it's
> reverse DNS. That is a local matter, delayed while "proving" we have
> ownership of the IP space to APNIC.

I see all your list messages so we're good to go.  Better to keep
discussion on list anyway so others can learn, participate, and it gets
archived.

> Thanks again for your advise.

You're welcome.

-- 
Stan

Re: RAID10 Performance

[Roberto Spadim]

> md/RAID1 doesn't guarantee symmetric read IO across members of the pair.
>  RAID1 isn't for high performance.  It's for cheap redundancy.  Even
> RAID0 can exhibit this behavior if you have a large chunk size and lots
> of small files.  They tend to align to the first drive in the stripe
> because they fit entirely within the chunk.  This is why a default 512KB
> chunk is insane for most workloads, especially mail servers.

raid1 is very very good for parallel read (many threads reading different files)
in my tests a high load database server runs better in raid1 than
raid10 when it need server many clients in small queries instead big
queries and low users (here raid10 runs better)
i'm using harddisk not ssd

some news in raid1 read algorithm are promissing and i think we will
have betters numbers in tests...


-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial

Re: RAID10 Performance

[Adam Goryachev]

On 09/08/12 02:59, Stan Hoeppner wrote:
> On 8/7/2012 10:49 PM, Adam Goryachev wrote:
>> Just some followup questions, hopefully this isn't too off-topic for
>> this list, if it is please let me know.
> 
>> OK, with the given budget, we currently have a couple of options:
>> 1) Replace the primary SAN (which currently has 2 x 2TB WD Caviar Black
>> RE4 drives in RAID1 + a hot spare) with 5 x 1TB Raptors you suggested
>> above (4 x 1TB in RAID10 + 1 hot spare).
> 
> Wow, that is a tight budget.  You'll be increasing total IOPS by about
> 3x with 4 10k drives.  Probably not optimal for your workload but it's a
> good start.
> 
> Also, don't use a hot spare--just wastes ports, cages, dollars.  It's
> not needed with RAID10.  Keep a spare in a locked cabinet and hot swap
> the drives upon failure notification.  RAID10 has a very large "double
> drive failure kills array" window, like many months to years.
> 
> BTW, Caviar Black != RE4  They're different products.

Yes, well it's a brand new SAN solution, for some stupid reason I didn't
consider how this would perform differently compared to the previous SAN
(overland s1000 which was ... returned.... due to some issues
experienced there). So, yes, budget is somewhat restrictive at the moment.

>> 2) Replace the primary SAN with 3 x 480GB SSD drives in linear + one of
>> the existing 2TB drives combined as RAID1 with the 2TB drive in write
>> only mode. This reduces overall capacity, but it does provide enough
>> capacity for at least 6 to 12 months. If needed, one additional SSD will
>> provide almost 2TB across the entire san.
> 
> This is simply insane, frankly.  Don't attempt this, even if md does
> support a write only mirror partner.

OK, what if we manage to do 4 x SSD's providing 960GB space in RAID10,
this might be possible now, and then we can add additional SATA
controller with additional SSD's when we need to upgrade further.

> Wait until *enterprise* SSD is fully mature and less expensive.  Stick
> with mechanical storage for now as your budget doesn't support SSD.
> If/when you go SSD, go *all* SSD, not this asymmetric Frankenstein
> stuff, which will only cause you problems.

A slightly different question, is the reason you don't suggest SSD
because you feel that it is not as good as spinning disks (reliability
or something else?)

It would seem that SSD would be the ideal solution to this problem
(ignoring cost) in that it provides very high IOPS for random read/write
performance. I'm somewhat suggesting SSD as the best option, but I'm
starting to question that. I don't have a lot of experience with SSD's,
though my limited experience says they are perfectly good/fast/etc...

>> I'm aware that a single SSD failure will reduce performance back to
>> current levels.
> 
> Then why would you ever consider this?  This thread is about increasing
> performance.  Why would you build a system that will instantly decrease
> IOPS by a factor of 1000 upon device failure?  That's insane.

Well, poor performance during a hardware failure is acceptable... at
least, explainable, and long term may assist in getting additional
funding/upgrades. In any case, if we use 4 x SSD in RAID10, that should
avoid that issue, it is only if we need to failover to the secondary san
that performance will degrade (significantly).

>>> And don't use the default 512KB chunk size of metadata 1.2.  512KB per
>>> chunk is insane.  With your Win server VM workload, where no server does
>>> much writing of large files or at a sustained rate, usually only small
>>> files, you should be using a small chunk size, something like 32KB,
>>> maybe even 16KB.  If you use a large chunk size you'll rarely be able to
>>> fill a full stripe write, and you'll end up with IO hot spots on
>>> individual drives, decreasing performance.
>>
>> I'm assuming that this can't be changed?
> 
> You assume what can't be changed?  Define what is changing.  It's a
> simple command line switch to change the chunk size from the default.
> See man mdadm.

I meant can't be changed on the current MD, ie, convert the existing MD
device to a different chunk size.

>> Could I simply create a new MD array with the smaller chunk size, tell
> 
> So you're asking how to migrate from the current disks to the new disks?
>  Yes you obviously must create a new RAID10 array.  Do you have enough
> SAS/SATA ports to have all (2+4) disks running?  If so this should be
> straightforward but will require some downtime.
> 
>> DRBD to sync from the remote to this new array, and then do the same for
>> the other SAN?
> 
> Don't do this over ethernet.  What I would do is simply shut down all
> daemons that may write to the current array, make it "static".  Shut
> down DRBD on both hosts.  Use dd or a partition tool to copy everything
> from the 2TB md/RAID1 mirror to the new 4 drive array.  Change your
> mounts etc to the new RAID10 device.  Down the 2TB mirror array.
> Confirm the new array is working.  Delete the current DRBD configuration
> on both hosts and create a new one.  Start DRBD on both hosts and as its
> syncing up restart services.

We only have 5 available sata ports right now, so probably I will mostly
follow what you just said (only change is to create new array with one
missing disk, then after the dd, remove the two old drives, and add the
4th missing disk.

> md/RAID1 doesn't guarantee symmetric read IO across members of the pair.
>  RAID1 isn't for high performance.  It's for cheap redundancy.  

Actually, I always thought RAID1 was the most expensive RAID (since it
halves capacity) and provided the best read performance. Am I really
wrong :(

Why doesn't the md driver "attempt" to balance read requests across both
members of a RAID1? Or are you saying it does attempt to, it just isn't
guaranteed?

> Even RAID0 can exhibit this behavior if you have a large chunk size
> and lots of small files.  They tend to align to the first drive in
> the stripe because they fit entirely within the chunk.  This is why
> a default 512KB chunk is insane for most workloads, especially
> mail servers.

That is perfectly understandable on RAID0, since the data only exists in
one place, so you MUST read it from the disk it exists on. You are
optimizing how the data is spread by changing the chunk size/stripe
size/etc, not where it CAN be read from.

Finally, just to throw a really horrible thought into the mix... RAID5
is considered horrible because you need to read/modify/write when doing
a write smaller than the stripe size. Is this still a significant issue
when dealing with SSD's, where we don't care about the seek time to do
this? Or is RAID5 still silly to consider (I think it is)?

Thank you.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: RAID10 Performance

[Stan Hoeppner]

On 8/8/2012 8:00 PM, Adam Goryachev wrote:

> OK, what if we manage to do 4 x SSD's providing 960GB space in RAID10,
> this might be possible now, and then we can add additional SATA
> controller with additional SSD's when we need to upgrade further.

With SSD storage, latency goes to effectively zero and IOPS through the
roof.  Combined with the cost of SSD, parity RAID makes sense, even for
high IOPS workloads, as the RMW penalty is negligible.  So you'd want to
go with RAID5 and get 1.4TB of space.  The downside to this is md/RAID5
currently uses a single write thread, so under high IOPS load you'll
saturate one CPU core, and performance hits a brick wall, even if all
other cores are idle.  This is currently being addressed with various
patches in development.

> A slightly different question, is the reason you don't suggest SSD
> because you feel that it is not as good as spinning disks (reliability
> or something else?)

I don't suggest *consumer* SSDs for server workloads.  The 480GB units
you are looking at are consumer grade.

> It would seem that SSD would be the ideal solution to this problem
> (ignoring cost) in that it provides very high IOPS for random read/write
> performance. I'm somewhat suggesting SSD as the best option, but I'm
> starting to question that. I don't have a lot of experience with SSD's,
> though my limited experience says they are perfectly good/fast/etc...

Read about consumer vs enterprise grade SSD, status of Linux TRIM
support--block/filesystem layers, realtime vs batch discard, etc.

> I meant can't be changed on the current MD, ie, convert the existing MD
> device to a different chunk size.

Surely you already know the answer to this.

> We only have 5 available sata ports right now, so probably I will mostly
> follow what you just said (only change is to create new array with one
> missing disk, then after the dd, remove the two old drives, and add the
> 4th missing disk.

And do two massive data moving operations instead of one?  An array
build and a mirror sync instead of just an array build.  For this, and
other more important reasons, you should really get a new HBA for the 4
new Raptor drives.  The card plus one breakout cable runs $270 USD and
gives you 4 spare fast SAS/SATA ports for adding 4 more Raptor drives in
the future.  It's a bit faster than motherboard-down SATA ASICs in
general, and even more so under high IOPS/bandwidth workloads.

http://www.newegg.com/Product/Product.aspx?Item=N82E16816118112
http://www.newegg.com/Product/Product.aspx?Item=N82E16816116098

It also gives you the flexibility to keep the 2TB drives in the machine
for nearline/backup duty, etc, and leave 3 mobo ports available to
expand that.  You'll be much happier going this route.

> Actually, I always thought RAID1 was the most expensive RAID (since it
> halves capacity) and provided the best read performance. Am I really
> wrong :(

It's cheap because it only requires two drives.  All other RAID levels
require 3 or more, sans the quasi RAID configurations one or more of the
resident list idiots will surely retort with (rolls eyes).  Pure, i.e.
textbook original implementation, RAID1 read performance is the same as
a single drive.  md/RAID1 has a few tricks to increase read performance
on RAID1, but overall you won't get 2x read performance over a single
drive, not even close.

> Why doesn't the md driver "attempt" to balance read requests across both
> members of a RAID1? 

I'm not a kernel dev.  Ask Neil.

> Or are you saying it does attempt to, it just isn't
> guaranteed?

I was pretty clear.

> That is perfectly understandable on RAID0, since the data only exists in
> one place, so you MUST read it from the disk it exists on. You are
> optimizing how the data is spread by changing the chunk size/stripe
> size/etc, not where it CAN be read from.

You misunderstood my point.

> Finally, just to throw a really horrible thought into the mix... RAID5
> is considered horrible because you need to read/modify/write when doing
> a write smaller than the stripe size. 

This is true specifically of mechanical storage.  Creating a new stripe
with a partial width write is only one of multiple scenarios that will
cause an RMW.  In this case the RMW will occur later, when the
filesystem creates another small file(s) in the sectors of the stripe.
An RMW will occur immediately when modifying an existing file.

> Is this still a significant issue
> when dealing with SSD's, where we don't care about the seek time to do
> this? Or is RAID5 still silly to consider (I think it is)?

See up above.  Again, RAID5 is much more amenable to SSD due to the low
latency and high IOPS.  But with the current md/RAID5 single write
thread implementation, and a high write IOPS workload, you can easily
run out of CPU long before peaking the SSDs.  This is true of md/RAID
1/6/10 as well, but again is being addressed in development.  Currently
for maximum SSD write performance you need to use md/RAID0 or linear, as
both fully thread across all CPUs/cores.

-- 
Stan

Re: RAID10 Performance

[Mario 'BitKoenig' Holbe]

Aaron Sowry <aaron@cendio.se> wrote:
> 1) As I understand it, a RAID10 'near' configuration using two disks is
> essentially equivalent to a RAID1 configuration. Is this correct?

Yes.

> 2) Does md RAID1 support 'striped' reads? If not, is RAID1 read

No.

RAID1 stores it's data like:
disk1: ABCDEF
disk2: ABCDEF

You cannot stripe there. You can read A from disk1 and B from disk2, C
from disk1 etc. (and linux md does that - for bigger chunks: have a look
at read balancing), but you have to seek on both disks to skip the data
which has already been read from the other disk. And seeking on current
disks is approximately as fast or slow as reading for short distances.

> performance in any way related to the number of disks in the array?

Yes. The more disks you have in the array the more read()s can be served
parallel: One process can read A, another process can read C and both
reads can be served parallel without interfering each other. The more
disks you have the more reads can be served parallel.

> 3) From what I have read so far, a RAID10 'far' configuration on 2 disks
> provides increased read performance over an equivalent 'near'
> configuration, however I am struggling to understand exactly why. I
> understand the difference between the 'near' and 'far' configurations,
> but not *why* this should provide any speed increases. What am I missing?

See 2)
The 'far' configuration stores data in the first half of the disk like
RAID0:
disk1: ACE
disk2: BDF

Hence you can read the stream ABCDEF without seeking inbetween on any of
the disks.

> 4) I have performed a(n admittedly fairly basic) benchmark on the same
> system under two different configurations - RAID10,n2 and RAID10,f2
> using tiobench with default settings. In short, the results showed a
> significant speed increase for single-threaded sequential reads (83Mb/s
> vs 166MB/s),

... as one would expect.

> some increase for single-threaded random reads (1.85Mb/s vs
> 2.25Mb/s), but a decrease for every other metric, including
> multi-threaded sequential and random reads. I was expecting write

This does heavily depend on the size of the data you read and the chunk
size of the RAID10,f, and the location(s) of the data on disk.
Note: as you can see in 3) reading the stream ABCDEF utilizes two disks,
i.e. keeps them busy. What benefits a single process here will hurt
multiple processes: no other parallel reads can be served and reading
other data semi-parallel requires both disks to seek which decreases
performance.


regards
   Mario
-- 
There is nothing more deceptive than an obvious fact.
             -- Sherlock Holmes by Arthur Conan Doyle

Re: RAID10 Performance

[Mathias Burén]

On 2 March 2011 14:42, Mark Knecht <markknecht@gmail.com> wrote:
> On Wed, Mar 2, 2011 at 1:04 AM, Aaron Sowry <aaron@cendio.se> wrote:
> <SNIP>
>>
>> 2) Does md RAID1 support 'striped' reads? If not, is RAID1 read
>> performance in any way related to the number of disks in the array?
> <SNIP>
>
> RAID1 is all about redundancy/reliability and not about improving
> performance as measured by MB/S.
>
> - Mark
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


Sure,

But if you can get improved performance without impacting
redundancy/reliability, then why not?

Kind regards,
// Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID10 Performance

[Mark Knecht]

On Wed, Mar 2, 2011 at 1:04 AM, Aaron Sowry <aaron@cendio.se> wrote:
<SNIP>
>
> 2) Does md RAID1 support 'striped' reads? If not, is RAID1 read
> performance in any way related to the number of disks in the array?
<SNIP>

RAID1 is all about redundancy/reliability and not about improving
performance as measured by MB/S.

- Mark

Re: RAID10 Performance

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3105 bytes --]



On Wed, 02 Mar 2011 09:50:47 +0100 Aaron Sowry <aaron@cendio.se> wrote:

> Hello,
> 
> I have been testing different RAID configurations on a 2-disk setup, and
> have a couple of questions regarding performance. The information I have
> found online so far seems to contradict itself fairly regularly so I was
> hoping for a more coherent answer :)
> 
> 1) As I understand it, a RAID10 'near' configuration using two disks is
> essentially equivalent to a RAID1 configuration. Is this correct?

"equivalent" in that the data is stored in the same places.  However as
different code is used to access it, performances could be different.

> 
> 2) Does md RAID1 support 'striped' reads? If not, is RAID1 read
> performance in any way related to the number of disks in the array?

Not sure exactly what you mean by "striped" reads.  There is no concept of a
stripe in RAID1.
A single sequential read will tend to read from just one device.
Random reads will tend to use all devices.

> 
> 3) From what I have read so far, a RAID10 'far' configuration on 2 disks
> provides increased read performance over an equivalent 'near'
> configuration, however I am struggling to understand exactly why. I
> understand the difference between the 'near' and 'far' configurations,
> but not *why* this should provide any speed increases. What am I missing?

A sequential read on raid10-far2 will read from both devices in parallel
(once read-ahead kicks in or large enough reads are submitted).  So a single
sequential read from a RAID10-far2 should be (nearly) twice as fast as a
single sequential read from a RAID1 - as twice as many drives are used.

> 
> 4) I have performed a(n admittedly fairly basic) benchmark on the same
> system under two different configurations - RAID10,n2 and RAID10,f2
> using tiobench with default settings. In short, the results showed a
> significant speed increase for single-threaded sequential reads (83Mb/s
> vs 166MB/s), some increase for single-threaded random reads (1.85Mb/s vs
> 2.25Mb/s), but a decrease for every other metric, including
> multi-threaded sequential and random reads. I was expecting write
> performance to decrease slightly under RAID10,f2 compared to RAID10,n2,
> but am slightly confused about the multi-threaded read performance. Is
> it my expectations or my testing that needs to be reviewed?

If the  random reads are smaller than the chunk size and suitably aligned,
then you should get much the same performance - f2 might be a bit faster
because it always reads from the 'faster' region of the device.

If the random reads are larger than the chunk size, then each read will hit
both disks in f2, but only one disk in n2, so f2 will have more contention.

Writes (sequential or random) are certainly slower with f2 as the head keeps
seeking from one half of the device to the other.  Large chunk sizes will
reduce this a little.

What chunk size did you use?  What block size for random IO?
Do the results change as you vary these sizes?

NeilBrown


> 
> Cheers,
> Aaron
> 


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: RAID10 Performance

[Keld Jørn Simonsen]

On Wed, Mar 02, 2011 at 09:24:25AM +0000, Robin Hill wrote:
> On Wed Mar 02, 2011 at 10:04:19AM +0100, Aaron Sowry wrote:
> 
> > Hello,
> > 
> > I have been testing different RAID configurations on a 2-disk setup, and
> > have a couple of questions regarding performance. The information I have
> > found online so far seems to contradict itself fairly regularly so I was
> > hoping for a more coherent answer :)
> > 
> > 1) As I understand it, a RAID10 'near' configuration using two disks is
> > essentially equivalent to a RAID1 configuration. Is this correct?
> > 
> It should be, yes, though I've not actually sat down and verified this.
> 
> > 2) Does md RAID1 support 'striped' reads? If not, is RAID1 read
> > performance in any way related to the number of disks in the array?
> > 
> No, it doesn't support 'striped' reads (there's no real performance
> advantage to doing so as you'd be losing the time skipping between
> stripes that you'd gain on reading from multiple drives concurrently).
> You do get a performance advantage with multiple drives in that more
> requests can be handled concurrently though.
> 
> > 3) From what I have read so far, a RAID10 'far' configuration on 2 disks
> > provides increased read performance over an equivalent 'near'
> > configuration, however I am struggling to understand exactly why. I
> > understand the difference between the 'near' and 'far' configurations,
> > but not *why* this should provide any speed increases. What am I missing?
> > 
> The speed increase is because the read speed varies across a disk -
> reading from the outer sectors is faster than reading from the inner
> sectors. The 'far' configuration means that all data is available on the
> outer half of the drive, so reads should mostly be served from there.

Also for sequential reads you get something like double the performance,
compared to raid1. This is because the layout of "far" is the same as
raid0, so you get striping performance. 

> > 4) I have performed a(n admittedly fairly basic) benchmark on the same
> > system under two different configurations - RAID10,n2 and RAID10,f2
> > using tiobench with default settings. In short, the results showed a
> > significant speed increase for single-threaded sequential reads (83Mb/s
> > vs 166MB/s), some increase for single-threaded random reads (1.85Mb/s vs
> > 2.25Mb/s), but a decrease for every other metric, including
> > multi-threaded sequential and random reads. I was expecting write
> > performance to decrease under RAID10,f2 compared to RAID10,n2, but am
> > slightly confused about the multi-threaded read performance. Is it my
> > expectations or my testing that needs to be reviewed?
> > 
> I'm not sure about that one - I'd expect multi-threaded reads to be at
> least as good on a "far" layout as a "near", but I've not actually run
> any benchmarks myself.

I would also think multi-threaded sequential reads to be faster with
"far". Especially I would think that an additional sequential read on a
running file system would be much faster than with "near" or raid1. 
That is my experience with the systems I run here.

There are some figures on performance in:
https://raid.wiki.kernel.org/index.php/Performance
That indicates that the multi-threaded random reads are also better, 
in the order of 40 % better that "near", and 125 % better than raid1.
This was on a quite old kernel, but I don't think things have improved
much in this area in the kernel. It would be nice to also have your
numbers on the page.

best regards
keld

Re: RAID10 Performance

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 2815 bytes --]

On Wed Mar 02, 2011 at 10:04:19AM +0100, Aaron Sowry wrote:

> Hello,
> 
> I have been testing different RAID configurations on a 2-disk setup, and
> have a couple of questions regarding performance. The information I have
> found online so far seems to contradict itself fairly regularly so I was
> hoping for a more coherent answer :)
> 
> 1) As I understand it, a RAID10 'near' configuration using two disks is
> essentially equivalent to a RAID1 configuration. Is this correct?
> 
It should be, yes, though I've not actually sat down and verified this.

> 2) Does md RAID1 support 'striped' reads? If not, is RAID1 read
> performance in any way related to the number of disks in the array?
> 
No, it doesn't support 'striped' reads (there's no real performance
advantage to doing so as you'd be losing the time skipping between
stripes that you'd gain on reading from multiple drives concurrently).
You do get a performance advantage with multiple drives in that more
requests can be handled concurrently though.

> 3) From what I have read so far, a RAID10 'far' configuration on 2 disks
> provides increased read performance over an equivalent 'near'
> configuration, however I am struggling to understand exactly why. I
> understand the difference between the 'near' and 'far' configurations,
> but not *why* this should provide any speed increases. What am I missing?
> 
The speed increase is because the read speed varies across a disk -
reading from the outer sectors is faster than reading from the inner
sectors. The 'far' configuration means that all data is available on the
outer half of the drive, so reads should mostly be served from there.

> 4) I have performed a(n admittedly fairly basic) benchmark on the same
> system under two different configurations - RAID10,n2 and RAID10,f2
> using tiobench with default settings. In short, the results showed a
> significant speed increase for single-threaded sequential reads (83Mb/s
> vs 166MB/s), some increase for single-threaded random reads (1.85Mb/s vs
> 2.25Mb/s), but a decrease for every other metric, including
> multi-threaded sequential and random reads. I was expecting write
> performance to decrease under RAID10,f2 compared to RAID10,n2, but am
> slightly confused about the multi-threaded read performance. Is it my
> expectations or my testing that needs to be reviewed?
> 
I'm not sure about that one - I'd expect multi-threaded reads to be at
least as good on a "far" layout as a "near", but I've not actually run
any benchmarks myself.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

RAID10 Performance

[Aaron Sowry]

Hello,

I have been testing different RAID configurations on a 2-disk setup, and
have a couple of questions regarding performance. The information I have
found online so far seems to contradict itself fairly regularly so I was
hoping for a more coherent answer :)

1) As I understand it, a RAID10 'near' configuration using two disks is
essentially equivalent to a RAID1 configuration. Is this correct?

2) Does md RAID1 support 'striped' reads? If not, is RAID1 read
performance in any way related to the number of disks in the array?

3) From what I have read so far, a RAID10 'far' configuration on 2 disks
provides increased read performance over an equivalent 'near'
configuration, however I am struggling to understand exactly why. I
understand the difference between the 'near' and 'far' configurations,
but not *why* this should provide any speed increases. What am I missing?

4) I have performed a(n admittedly fairly basic) benchmark on the same
system under two different configurations - RAID10,n2 and RAID10,f2
using tiobench with default settings. In short, the results showed a
significant speed increase for single-threaded sequential reads (83Mb/s
vs 166MB/s), some increase for single-threaded random reads (1.85Mb/s vs
2.25Mb/s), but a decrease for every other metric, including
multi-threaded sequential and random reads. I was expecting write
performance to decrease under RAID10,f2 compared to RAID10,n2, but am
slightly confused about the multi-threaded read performance. Is it my
expectations or my testing that needs to be reviewed?

Cheers,
Aaron

RAID10 Performance

[Aaron Sowry]

[-- Attachment #1: Type: text/plain, Size: 1617 bytes --]

Hello,

I have been testing different RAID configurations on a 2-disk setup, and
have a couple of questions regarding performance. The information I have
found online so far seems to contradict itself fairly regularly so I was
hoping for a more coherent answer :)

1) As I understand it, a RAID10 'near' configuration using two disks is
essentially equivalent to a RAID1 configuration. Is this correct?

2) Does md RAID1 support 'striped' reads? If not, is RAID1 read
performance in any way related to the number of disks in the array?

3) From what I have read so far, a RAID10 'far' configuration on 2 disks
provides increased read performance over an equivalent 'near'
configuration, however I am struggling to understand exactly why. I
understand the difference between the 'near' and 'far' configurations,
but not *why* this should provide any speed increases. What am I missing?

4) I have performed a(n admittedly fairly basic) benchmark on the same
system under two different configurations - RAID10,n2 and RAID10,f2
using tiobench with default settings. In short, the results showed a
significant speed increase for single-threaded sequential reads (83Mb/s
vs 166MB/s), some increase for single-threaded random reads (1.85Mb/s vs
2.25Mb/s), but a decrease for every other metric, including
multi-threaded sequential and random reads. I was expecting write
performance to decrease slightly under RAID10,f2 compared to RAID10,n2,
but am slightly confused about the multi-threaded read performance. Is
it my expectations or my testing that needs to be reviewed?

Cheers,
Aaron


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]