* Performance question, RAID5 @ 2011-01-29 22:48 Mathias Burén 2011-01-29 22:53 ` Roman Mamedov 2011-01-29 23:26 ` CoolCold 0 siblings, 2 replies; 37+ messages in thread From: Mathias Burén @ 2011-01-29 22:48 UTC (permalink / raw) To: Linux-RAID Hi, I'm wondering if the performance I'm getting is OK or if there's something I can do about it. Also, where the potential bottlenecks are. Setup: 6x2TB HDDs, their performance: /dev/sdb: Timing cached reads: 1322 MB in 2.00 seconds = 661.51 MB/sec Timing buffered disk reads: 362 MB in 3.02 seconds = 120.06 MB/sec /dev/sdc: Timing cached reads: 1282 MB in 2.00 seconds = 641.20 MB/sec Timing buffered disk reads: 342 MB in 3.01 seconds = 113.53 MB/sec /dev/sdd: Timing cached reads: 1282 MB in 2.00 seconds = 640.55 MB/sec Timing buffered disk reads: 344 MB in 3.00 seconds = 114.58 MB/sec /dev/sde: Timing cached reads: 1328 MB in 2.00 seconds = 664.46 MB/sec Timing buffered disk reads: 350 MB in 3.01 seconds = 116.37 MB/sec /dev/sdf: Timing cached reads: 1304 MB in 2.00 seconds = 651.55 MB/sec Timing buffered disk reads: 378 MB in 3.01 seconds = 125.62 MB/sec /dev/sdg: Timing cached reads: 1324 MB in 2.00 seconds = 661.91 MB/sec Timing buffered disk reads: 400 MB in 3.00 seconds = 133.15 MB/sec These are used in a RAID5 setup: Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdf1[0] sdg1[6] sde1[5] sdc1[3] sdd1[4] sdb1[1] 9751756800 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU] unused devices: <none> /dev/md0: Version : 1.2 Creation Time : Tue Oct 19 08:58:41 2010 Raid Level : raid5 Array Size : 9751756800 (9300.00 GiB 9985.80 GB) Used Dev Size : 1950351360 (1860.00 GiB 1997.16 GB) Raid Devices : 6 Total Devices : 6 Persistence : Superblock is persistent Update Time : Fri Jan 28 14:55:48 2011 State : clean Active Devices : 6 Working Devices : 6 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K Name : ion:0 (local to host ion) UUID : e6595c64:b3ae90b3:f01133ac:3f402d20 Events : 3035769 Number Major Minor RaidDevice State 0 8 81 0 active sync /dev/sdf1 1 8 17 1 active sync /dev/sdb1 4 8 49 2 active sync /dev/sdd1 3 8 33 3 active sync /dev/sdc1 5 8 65 4 active sync /dev/sde1 6 8 97 5 active sync /dev/sdg1 As you can see they are partitioned. They are all identical like this: Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes 81 heads, 63 sectors/track, 765633 cylinders, total 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0e5b3a7a Device Boot Start End Blocks Id System /dev/sdb1 2048 3907029167 1953513560 fd Linux raid autodetect On this I run LVM: --- Physical volume --- PV Name /dev/md0 VG Name lvstorage PV Size 9.08 TiB / not usable 1.00 MiB Allocatable yes (but full) PE Size 1.00 MiB Total PE 9523199 Free PE 0 Allocated PE 9523199 PV UUID YLEUKB-klxF-X3gF-6dG3-DL4R-xebv-6gKQc2 On top of the LVM I have: --- Volume group --- VG Name lvstorage System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 6 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 9.08 TiB PE Size 1.00 MiB Total PE 9523199 Alloc PE / Size 9523199 / 9.08 TiB Free PE / Size 0 / 0 VG UUID Xd0HTM-azdN-v9kJ-C7vD-COcU-Cnn8-6AJ6hI And in turn: --- Logical volume --- LV Name /dev/lvstorage/storage VG Name lvstorage LV UUID 9wsJ0u-0QMs-lL5h-E2UA-7QJa-l46j-oWkSr3 LV Write Access read/write LV Status available # open 1 LV Size 9.08 TiB Current LE 9523199 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 1280 Block device 254:1 And on that (sorry) there's the ext4 partition: /dev/mapper/lvstorage-storage on /raid5volume type ext4 (rw,noatime,barrier=1,nouser_xattr) Here are the numbers: /raid5volume $ time dd if=/dev/zero of=./bigfile.tmp bs=1M count=8192 8192+0 records in 8192+0 records out 8589934592 bytes (8.6 GB) copied, 94.0967 s, 91.3 MB/s real 1m34.102s user 0m0.107s sys 0m54.693s /raid5volume $ time dd if=./bigfile.tmp of=/dev/null bs=1M 8192+0 records in 8192+0 records out 8589934592 bytes (8.6 GB) copied, 37.8557 s, 227 MB/s real 0m37.861s user 0m0.053s sys 0m23.608s I saw that the process md0_raid5 spike sometimes on CPU usage. This is an Atom @ 1.6GHz, is that what is limiting the results? Here's bonnie++: /raid5volume/temp $ time bonnie++ -d ./ -m ion Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03e ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP ion 7G 13726 98 148051 87 68020 41 14547 99 286647 61 404.1 2 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 20707 99 +++++ +++ 25870 99 21242 98 +++++ +++ 25630 100 ion,7G,13726,98,148051,87,68020,41,14547,99,286647,61,404.1,2,16,20707,99,+++++,+++,25870,99,21242,98,+++++,+++,25630,100 real 20m54.320s user 16m10.447s sys 2m45.543s Thanks in advance, // Mathias ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-29 22:48 Performance question, RAID5 Mathias Burén @ 2011-01-29 22:53 ` Roman Mamedov 2011-01-29 23:44 ` Mathias Burén 2011-01-29 23:26 ` CoolCold 1 sibling, 1 reply; 37+ messages in thread From: Roman Mamedov @ 2011-01-29 22:53 UTC (permalink / raw) To: Mathias Burén; +Cc: Linux-RAID [-- Attachment #1: Type: text/plain, Size: 410 bytes --] On Sat, 29 Jan 2011 22:48:06 +0000 Mathias Burén <mathias.buren@gmail.com> wrote: > Hi, > > I'm wondering if the performance I'm getting is OK or if there's > something I can do about it. Also, where the potential bottlenecks > are. How are your disks plugged in? Which controller model(s), which bus. But generally, on an Atom 1.6 Ghz those seem like good results. -- With respect, Roman [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-29 22:53 ` Roman Mamedov @ 2011-01-29 23:44 ` Mathias Burén 2011-01-29 23:57 ` Roman Mamedov 2011-01-30 1:52 ` Keld Jørn Simonsen 0 siblings, 2 replies; 37+ messages in thread From: Mathias Burén @ 2011-01-29 23:44 UTC (permalink / raw) To: Roman Mamedov; +Cc: Linux-RAID On 29 January 2011 22:53, Roman Mamedov <rm@romanrm.ru> wrote: > On Sat, 29 Jan 2011 22:48:06 +0000 > Mathias Burén <mathias.buren@gmail.com> wrote: > >> Hi, >> >> I'm wondering if the performance I'm getting is OK or if there's >> something I can do about it. Also, where the potential bottlenecks >> are. > > How are your disks plugged in? Which controller model(s), which bus. > But generally, on an Atom 1.6 Ghz those seem like good results. > > -- > With respect, > Roman > Hi, Sorry, of course I should've included that. Here's the info: ~/bin $ sudo ./drivescan.sh Controller device @ pci0000:00/0000:00:0b.0 [ahci] SATA controller: nVidia Corporation MCP79 AHCI Controller (rev b1) host0: /dev/sda ATA Corsair CSSD-F60 {SN: 10326505580009990027} host1: /dev/sdb ATA WDC WD20EARS-00M {SN: WD-WCAZA1022443} host2: /dev/sdc ATA WDC WD20EARS-00M {SN: WD-WMAZ20152590} host3: /dev/sdd ATA WDC WD20EARS-00M {SN: WD-WMAZ20188479} host4: [Empty] host5: [Empty] Controller device @ pci0000:00/0000:00:16.0/0000:05:00.0 [sata_mv] SCSI storage controller: HighPoint Technologies, Inc. RocketRAID 230x 4 Port SATA-II Controller (rev 02) host6: [Empty] host7: /dev/sde ATA SAMSUNG HD204UI {SN: S2HGJ1RZ800964 } host8: /dev/sdf ATA WDC WD20EARS-00M {SN: WD-WCAZA1000331} host9: /dev/sdg ATA SAMSUNG HD204UI {SN: S2HGJ1RZ800850 } It's all SATA 3Gbs. OK, so from what you're saying I should see significantly better results on a better CPU? The HDDs should be able to push 80MB/s (read or write), and that should yield at least 5*80 = 400MB/s (-1 for parity) on easy (sequential?) reads. // Mathias -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-29 23:44 ` Mathias Burén @ 2011-01-29 23:57 ` Roman Mamedov 2011-01-30 0:15 ` Stan Hoeppner 2011-01-30 0:27 ` Mathias Burén 2011-01-30 1:52 ` Keld Jørn Simonsen 1 sibling, 2 replies; 37+ messages in thread From: Roman Mamedov @ 2011-01-29 23:57 UTC (permalink / raw) To: Mathias Burén; +Cc: Linux-RAID [-- Attachment #1: Type: text/plain, Size: 1337 bytes --] On Sat, 29 Jan 2011 23:44:01 +0000 Mathias Burén <mathias.buren@gmail.com> wrote: > Controller device @ pci0000:00/0000:00:16.0/0000:05:00.0 [sata_mv] > SCSI storage controller: HighPoint Technologies, Inc. RocketRAID > 230x 4 Port SATA-II Controller (rev 02) > host6: [Empty] > host7: /dev/sde ATA SAMSUNG HD204UI {SN: S2HGJ1RZ800964 } > host8: /dev/sdf ATA WDC WD20EARS-00M {SN: WD-WCAZA1000331} > host9: /dev/sdg ATA SAMSUNG HD204UI {SN: S2HGJ1RZ800850 } Does this controller support PCI-E 2.0? I doubt it. Does you Atom mainboard support PCI-E 2.0? I highly doubt it. And if PCI-E 1.0/1.1 is used, these last 3 drives are limited to 250 MB/sec. in total, which in reality will be closer to 200 MB/sec. > It's all SATA 3Gbs. OK, so from what you're saying I should see > significantly better results on a better CPU? The HDDs should be able > to push 80MB/s (read or write), and that should yield at least 5*80 = > 400MB/s (-1 for parity) on easy (sequential?) reads. According to the hdparm benchmark, your CPU can not read faster than 640 MB/sec from _RAM_, and that's just plain easy linear data from a buffer. So it is perhaps not promising with regard to whether you will get 400MB/sec reading from RAID6 (with all the corresponding overheads) or not. -- With respect, Roman [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-29 23:57 ` Roman Mamedov @ 2011-01-30 0:15 ` Stan Hoeppner 2011-01-30 0:33 ` Mathias Burén 2011-01-30 0:27 ` Mathias Burén 1 sibling, 1 reply; 37+ messages in thread From: Stan Hoeppner @ 2011-01-30 0:15 UTC (permalink / raw) To: Roman Mamedov; +Cc: Mathias Burén, Linux-RAID Roman Mamedov put forth on 1/29/2011 5:57 PM: > On Sat, 29 Jan 2011 23:44:01 +0000 > Mathias Burén <mathias.buren@gmail.com> wrote: > >> Controller device @ pci0000:00/0000:00:16.0/0000:05:00.0 [sata_mv] >> SCSI storage controller: HighPoint Technologies, Inc. RocketRAID >> 230x 4 Port SATA-II Controller (rev 02) >> host6: [Empty] >> host7: /dev/sde ATA SAMSUNG HD204UI {SN: S2HGJ1RZ800964 } >> host8: /dev/sdf ATA WDC WD20EARS-00M {SN: WD-WCAZA1000331} >> host9: /dev/sdg ATA SAMSUNG HD204UI {SN: S2HGJ1RZ800850 } > > Does this controller support PCI-E 2.0? I doubt it. > Does you Atom mainboard support PCI-E 2.0? I highly doubt it. > And if PCI-E 1.0/1.1 is used, these last 3 drives are limited to 250 MB/sec. > in total, which in reality will be closer to 200 MB/sec. > >> It's all SATA 3Gbs. OK, so from what you're saying I should see >> significantly better results on a better CPU? The HDDs should be able >> to push 80MB/s (read or write), and that should yield at least 5*80 = >> 400MB/s (-1 for parity) on easy (sequential?) reads. > > According to the hdparm benchmark, your CPU can not read faster than 640 > MB/sec from _RAM_, and that's just plain easy linear data from a buffer. So it > is perhaps not promising with regard to whether you will get 400MB/sec reading > from RAID6 (with all the corresponding overheads) or not. It's also not promising given that 4 of his 6 drives are WDC-WD20EARS, which suck harder than a Dirt Devil at 240 volts, and the fact his 6 drives don't match. Sure, you say "Non matching drives are what software RAID is for right?" Wrong, if you want best performance. About the only things that might give you a decent boost at this point are some EXT4 mount options in /etc/fstab: data=writeback,barrier=0 The first eliminates strict write ordering. The second disables write barriers, so the drive's caches don't get flushed by Linux, and instead work as the firmware intends. The first of these is safe. The second may cause some additional data loss if writes are in flight when the power goes out or the kernel crashes. I'd recommend adding both to fstab, reboot and run your tests. Post the results here. If you have a decent UPS and auto shutdown software to down the system when the battery gets low during an outage, keep these settings if they yield substantially better performance. -- Stan -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 0:15 ` Stan Hoeppner @ 2011-01-30 0:33 ` Mathias Burén 0 siblings, 0 replies; 37+ messages in thread From: Mathias Burén @ 2011-01-30 0:33 UTC (permalink / raw) To: Stan Hoeppner; +Cc: Roman Mamedov, Linux-RAID On 30 January 2011 00:15, Stan Hoeppner <stan@hardwarefreak.com> wrote: > Roman Mamedov put forth on 1/29/2011 5:57 PM: >> On Sat, 29 Jan 2011 23:44:01 +0000 >> Mathias Burén <mathias.buren@gmail.com> wrote: >> >>> Controller device @ pci0000:00/0000:00:16.0/0000:05:00.0 [sata_mv] >>> SCSI storage controller: HighPoint Technologies, Inc. RocketRAID >>> 230x 4 Port SATA-II Controller (rev 02) >>> host6: [Empty] >>> host7: /dev/sde ATA SAMSUNG HD204UI {SN: S2HGJ1RZ800964 } >>> host8: /dev/sdf ATA WDC WD20EARS-00M {SN: WD-WCAZA1000331} >>> host9: /dev/sdg ATA SAMSUNG HD204UI {SN: S2HGJ1RZ800850 } >> >> Does this controller support PCI-E 2.0? I doubt it. >> Does you Atom mainboard support PCI-E 2.0? I highly doubt it. >> And if PCI-E 1.0/1.1 is used, these last 3 drives are limited to 250 MB/sec. >> in total, which in reality will be closer to 200 MB/sec. >> >>> It's all SATA 3Gbs. OK, so from what you're saying I should see >>> significantly better results on a better CPU? The HDDs should be able >>> to push 80MB/s (read or write), and that should yield at least 5*80 = >>> 400MB/s (-1 for parity) on easy (sequential?) reads. >> >> According to the hdparm benchmark, your CPU can not read faster than 640 >> MB/sec from _RAM_, and that's just plain easy linear data from a buffer. So it >> is perhaps not promising with regard to whether you will get 400MB/sec reading >> from RAID6 (with all the corresponding overheads) or not. > > It's also not promising given that 4 of his 6 drives are WDC-WD20EARS, which > suck harder than a Dirt Devil at 240 volts, and the fact his 6 drives don't > match. Sure, you say "Non matching drives are what software RAID is for right?" > Wrong, if you want best performance. > > About the only things that might give you a decent boost at this point are some > EXT4 mount options in /etc/fstab: data=writeback,barrier=0 > > The first eliminates strict write ordering. The second disables write barriers, > so the drive's caches don't get flushed by Linux, and instead work as the > firmware intends. The first of these is safe. The second may cause some > additional data loss if writes are in flight when the power goes out or the > kernel crashes. I'd recommend adding both to fstab, reboot and run your tests. > Post the results here. > > If you have a decent UPS and auto shutdown software to down the system when the > battery gets low during an outage, keep these settings if they yield > substantially better performance. > > -- > Stan > Right. I wasn't using the writeback option. I won't disable barriers as I've no UPS. I've seen the stripe= ext4 mount option, from http://www.mjmwired.net/kernel/Documentation/filesystems/ext4.txt : 287 stripe=n Number of filesystem blocks that mballoc will try 288 to use for allocation size and alignment. For RAID5/6 289 systems this should be the number of data 290 disks * RAID chunk size in file system blocks. I suppose in my case, number of data disks is 5, RAID chunk size is 64KB, file system block size is 4KB. This is on top of LVM, I don't know how that affects the situation. So, the mount option would be stripe=80? (5*64/4) // Mathias -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-29 23:57 ` Roman Mamedov 2011-01-30 0:15 ` Stan Hoeppner @ 2011-01-30 0:27 ` Mathias Burén 1 sibling, 0 replies; 37+ messages in thread From: Mathias Burén @ 2011-01-30 0:27 UTC (permalink / raw) To: Roman Mamedov; +Cc: Linux-RAID On 29 January 2011 23:57, Roman Mamedov <rm@romanrm.ru> wrote: > On Sat, 29 Jan 2011 23:44:01 +0000 > Mathias Burén <mathias.buren@gmail.com> wrote: > >> Controller device @ pci0000:00/0000:00:16.0/0000:05:00.0 [sata_mv] >> SCSI storage controller: HighPoint Technologies, Inc. RocketRAID >> 230x 4 Port SATA-II Controller (rev 02) >> host6: [Empty] >> host7: /dev/sde ATA SAMSUNG HD204UI {SN: S2HGJ1RZ800964 } >> host8: /dev/sdf ATA WDC WD20EARS-00M {SN: WD-WCAZA1000331} >> host9: /dev/sdg ATA SAMSUNG HD204UI {SN: S2HGJ1RZ800850 } > > Does this controller support PCI-E 2.0? I doubt it. > Does you Atom mainboard support PCI-E 2.0? I highly doubt it. > And if PCI-E 1.0/1.1 is used, these last 3 drives are limited to 250 MB/sec. > in total, which in reality will be closer to 200 MB/sec. > >> It's all SATA 3Gbs. OK, so from what you're saying I should see >> significantly better results on a better CPU? The HDDs should be able >> to push 80MB/s (read or write), and that should yield at least 5*80 = >> 400MB/s (-1 for parity) on easy (sequential?) reads. > > According to the hdparm benchmark, your CPU can not read faster than 640 > MB/sec from _RAM_, and that's just plain easy linear data from a buffer. So it > is perhaps not promising with regard to whether you will get 400MB/sec reading > from RAID6 (with all the corresponding overheads) or not. > > -- > With respect, > Roman > Ah, right. The Ion platform actually supports PCI-E 2.0, but the controller I'm using doesn't, according to lspci, if I understand it correctly. SATA ontroller: 05:00.0 SCSI storage controller: HighPoint Technologies, Inc. RocketRAID 230x 4 Port SATA-II Controller (rev 02) Subsystem: Marvell Technology Group Ltd. Device 11ab [....] Capabilities: [60] Express (v1) Legacy Endpoint, MSI 00 PCI-Express bridge: 00:18.0 PCI bridge: nVidia Corporation MCP79 PCI Express Bridge (rev b1) (prog-if 00 [Normal decode]) Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- [...] Capabilities: [80] Express (v2) Root Port (Slot+), MSI 00 That might explain why the different stripe caches didn't have any effect either. Thanks for pointing that out, apparently I didn't think about it when I purchased the (super cheap) card. // Mathias -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-29 23:44 ` Mathias Burén 2011-01-29 23:57 ` Roman Mamedov @ 2011-01-30 1:52 ` Keld Jørn Simonsen 2011-01-30 1:54 ` Mathias Burén 1 sibling, 1 reply; 37+ messages in thread From: Keld Jørn Simonsen @ 2011-01-30 1:52 UTC (permalink / raw) To: Mathias Burén; +Cc: Roman Mamedov, Linux-RAID On Sat, Jan 29, 2011 at 11:44:01PM +0000, Mathias Burén wrote: > On 29 January 2011 22:53, Roman Mamedov <rm@romanrm.ru> wrote: > > On Sat, 29 Jan 2011 22:48:06 +0000 > > Mathias Burén <mathias.buren@gmail.com> wrote: > > > >> Hi, > >> > >> I'm wondering if the performance I'm getting is OK or if there's > >> something I can do about it. Also, where the potential bottlenecks > >> are. > > > > How are your disks plugged in? Which controller model(s), which bus. > > But generally, on an Atom 1.6 Ghz those seem like good results. > > > > -- > > With respect, > > Roman > > > > > Hi, > > Sorry, of course I should've included that. Here's the info: > > ~/bin $ sudo ./drivescan.sh > Controller device @ pci0000:00/0000:00:0b.0 [ahci] > SATA controller: nVidia Corporation MCP79 AHCI Controller (rev b1) > host0: /dev/sda ATA Corsair CSSD-F60 {SN: 10326505580009990027} > host1: /dev/sdb ATA WDC WD20EARS-00M {SN: WD-WCAZA1022443} > host2: /dev/sdc ATA WDC WD20EARS-00M {SN: WD-WMAZ20152590} > host3: /dev/sdd ATA WDC WD20EARS-00M {SN: WD-WMAZ20188479} > host4: [Empty] > host5: [Empty] Hmm, it seems like you have 2 empty slots on the on-board SATA controller. Try to move 2 of the disks from the other controller to the on-board controller. And I would also avoid LVM. I think LVM affects striping. best regards Keld -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 1:52 ` Keld Jørn Simonsen @ 2011-01-30 1:54 ` Mathias Burén 2011-01-30 5:56 ` Keld Jørn Simonsen 0 siblings, 1 reply; 37+ messages in thread From: Mathias Burén @ 2011-01-30 1:54 UTC (permalink / raw) To: Keld Jørn Simonsen; +Cc: Linux-RAID On 30 January 2011 01:52, Keld Jørn Simonsen <keld@keldix.com> wrote: > On Sat, Jan 29, 2011 at 11:44:01PM +0000, Mathias Burén wrote: >> On 29 January 2011 22:53, Roman Mamedov <rm@romanrm.ru> wrote: >> > On Sat, 29 Jan 2011 22:48:06 +0000 >> > Mathias Burén <mathias.buren@gmail.com> wrote: >> > >> >> Hi, >> >> >> >> I'm wondering if the performance I'm getting is OK or if there's >> >> something I can do about it. Also, where the potential bottlenecks >> >> are. >> > >> > How are your disks plugged in? Which controller model(s), which bus. >> > But generally, on an Atom 1.6 Ghz those seem like good results. >> > >> > -- >> > With respect, >> > Roman >> > >> >> >> Hi, >> >> Sorry, of course I should've included that. Here's the info: >> >> ~/bin $ sudo ./drivescan.sh >> Controller device @ pci0000:00/0000:00:0b.0 [ahci] >> SATA controller: nVidia Corporation MCP79 AHCI Controller (rev b1) >> host0: /dev/sda ATA Corsair CSSD-F60 {SN: 10326505580009990027} >> host1: /dev/sdb ATA WDC WD20EARS-00M {SN: WD-WCAZA1022443} >> host2: /dev/sdc ATA WDC WD20EARS-00M {SN: WD-WMAZ20152590} >> host3: /dev/sdd ATA WDC WD20EARS-00M {SN: WD-WMAZ20188479} >> host4: [Empty] >> host5: [Empty] > > > Hmm, it seems like you have 2 empty slots on the on-board SATA > controller. Try to move 2 of the disks from the other controller to the > on-board controller. > > And I would also avoid LVM. I think LVM affects striping. > > best regards > Keld > Sadly the 2 empty slots are not to be found on the motherboard, I guess they're in the chipset only. // Mathias -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 1:54 ` Mathias Burén @ 2011-01-30 5:56 ` Keld Jørn Simonsen 2011-01-30 12:12 ` Mathias Burén 0 siblings, 1 reply; 37+ messages in thread From: Keld Jørn Simonsen @ 2011-01-30 5:56 UTC (permalink / raw) To: Mathias Burén; +Cc: Keld Jørn Simonsen, Linux-RAID On Sun, Jan 30, 2011 at 01:54:42AM +0000, Mathias Burén wrote: > On 30 January 2011 01:52, Keld Jørn Simonsen <keld@keldix.com> wrote: > > On Sat, Jan 29, 2011 at 11:44:01PM +0000, Mathias Burén wrote: > >> On 29 January 2011 22:53, Roman Mamedov <rm@romanrm.ru> wrote: > >> > On Sat, 29 Jan 2011 22:48:06 +0000 > >> > Mathias Burén <mathias.buren@gmail.com> wrote: > >> > > >> >> Hi, > >> >> > >> >> I'm wondering if the performance I'm getting is OK or if there's > >> >> something I can do about it. Also, where the potential bottlenecks > >> >> are. > >> > > >> > How are your disks plugged in? Which controller model(s), which bus. > >> > But generally, on an Atom 1.6 Ghz those seem like good results. > >> > > >> > -- > >> > With respect, > >> > Roman > >> > > >> > >> > >> Hi, > >> > >> Sorry, of course I should've included that. Here's the info: > >> > >> ~/bin $ sudo ./drivescan.sh > >> Controller device @ pci0000:00/0000:00:0b.0 [ahci] > >> SATA controller: nVidia Corporation MCP79 AHCI Controller (rev b1) > >> host0: /dev/sda ATA Corsair CSSD-F60 {SN: 10326505580009990027} > >> host1: /dev/sdb ATA WDC WD20EARS-00M {SN: WD-WCAZA1022443} > >> host2: /dev/sdc ATA WDC WD20EARS-00M {SN: WD-WMAZ20152590} > >> host3: /dev/sdd ATA WDC WD20EARS-00M {SN: WD-WMAZ20188479} > >> host4: [Empty] > >> host5: [Empty] > > > > > > Hmm, it seems like you have 2 empty slots on the on-board SATA > > controller. Try to move 2 of the disks from the other controller to the > > on-board controller. > > > > And I would also avoid LVM. I think LVM affects striping. > > > > best regards > > Keld > > > > Sadly the 2 empty slots are not to be found on the motherboard, I > guess they're in the chipset only. maybe then use the 5th drive on the sata on-board controller in the raid5 - the sda drive. If the raid5 is where you want performance from. what do you use sda for? Your OS? It is a lot of space to use just for the OS. It could easily go into the raid5 too. And you culd use a raid for the system too, to secure you from bad things happening to your system. Or you could have a few 5 to 10 GB partitions in the beginning of each drive, for experimenting with raid layout and performance. This should be outside any LVM to exclude LVM having an impact on the tests. Maybe your PCI-e cannot do more than 2.5 Gbit - then 2 of your disks would be enough to fill that connection. You could try out a raid0 on the 3 drives. If you cannot get more than about 300 MB/s, then the PCI-E is a bottleneck. If that is so, then having 3 drives from the PCI-E could slow down the whole raid5, and using only 2 drives could speed up the full raid5. The on-board sata controller is normally much faster, having a direct connection to the southbridge - and typically a speed in the neighbourhood of 20 Gbit - or 2500 MB/s - which would be enough for many systems to not be the bottleneck. It can often pay off to have a motherboard with two on-board sata controllers with in total 8 SATA ports or more, instead of bying an extra PCI-E controller. Looking forward to hear what you find out. best regards keld -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 5:56 ` Keld Jørn Simonsen @ 2011-01-30 12:12 ` Mathias Burén 2011-01-30 19:44 ` Stan Hoeppner 2011-02-01 11:37 ` John Robinson 0 siblings, 2 replies; 37+ messages in thread From: Mathias Burén @ 2011-01-30 12:12 UTC (permalink / raw) To: Keld Jørn Simonsen; +Cc: Linux-RAID 2011/1/30 Keld Jørn Simonsen <keld@keldix.com>: > On Sun, Jan 30, 2011 at 01:54:42AM +0000, Mathias Burén wrote: >> On 30 January 2011 01:52, Keld Jørn Simonsen <keld@keldix.com> wrote: >> > On Sat, Jan 29, 2011 at 11:44:01PM +0000, Mathias Burén wrote: >> >> On 29 January 2011 22:53, Roman Mamedov <rm@romanrm.ru> wrote: >> >> > On Sat, 29 Jan 2011 22:48:06 +0000 >> >> > Mathias Burén <mathias.buren@gmail.com> wrote: >> >> > >> >> >> Hi, >> >> >> >> >> >> I'm wondering if the performance I'm getting is OK or if there's >> >> >> something I can do about it. Also, where the potential bottlenecks >> >> >> are. >> >> > >> >> > How are your disks plugged in? Which controller model(s), which bus. >> >> > But generally, on an Atom 1.6 Ghz those seem like good results. >> >> > >> >> > -- >> >> > With respect, >> >> > Roman >> >> > >> >> >> >> >> >> Hi, >> >> >> >> Sorry, of course I should've included that. Here's the info: >> >> >> >> ~/bin $ sudo ./drivescan.sh >> >> Controller device @ pci0000:00/0000:00:0b.0 [ahci] >> >> SATA controller: nVidia Corporation MCP79 AHCI Controller (rev b1) >> >> host0: /dev/sda ATA Corsair CSSD-F60 {SN: 10326505580009990027} >> >> host1: /dev/sdb ATA WDC WD20EARS-00M {SN: WD-WCAZA1022443} >> >> host2: /dev/sdc ATA WDC WD20EARS-00M {SN: WD-WMAZ20152590} >> >> host3: /dev/sdd ATA WDC WD20EARS-00M {SN: WD-WMAZ20188479} >> >> host4: [Empty] >> >> host5: [Empty] >> > >> > >> > Hmm, it seems like you have 2 empty slots on the on-board SATA >> > controller. Try to move 2 of the disks from the other controller to the >> > on-board controller. >> > >> > And I would also avoid LVM. I think LVM affects striping. >> > >> > best regards >> > Keld >> > >> >> Sadly the 2 empty slots are not to be found on the motherboard, I >> guess they're in the chipset only. > > maybe then use the 5th drive on the sata on-board controller in the > raid5 - the sda drive. If the raid5 is where you want performance from. > > what do you use sda for? Your OS? It is a lot of space to use just for > the OS. It could easily go into the raid5 too. And you culd use a raid > for the system too, to secure you from bad things happening to your > system. > > Or you could have a few 5 to 10 GB partitions in the beginning of > each drive, for experimenting with raid layout and performance. > This should be outside any LVM to exclude LVM having an impact on > the tests. > > Maybe your PCI-e cannot do more than 2.5 Gbit - then 2 of your disks > would be enough to fill that connection. You could try out > a raid0 on the 3 drives. If you cannot get more than about 300 MB/s, then > the PCI-E is a bottleneck. > > If that is so, then having 3 drives from the PCI-E could slow down the > whole raid5, and using only 2 drives could speed up the full raid5. > > The on-board sata controller is normally much faster, having a direct > connection to the southbridge - and typically a speed in the neighbourhood of > 20 Gbit - or 2500 MB/s - which would be enough for many systems to not > be the bottleneck. It can often pay off to have a motherboard with two > on-board sata controllers with in total 8 SATA ports or more, > instead of bying an extra PCI-E controller. > > Looking forward to hear what you find out. > > best regards > keld > Ah, good point. The sda is a 60GB SSD, I should definitely move that to the PCI-E card, as it doesn't do heavy IO (just small random r/w). Then i can have 4 RAID HDDs on the onboard ctrl, and 2 on the PCI-E shared with the SSD. If the SSD is idle then I should get ideal throughputs. Thanks to mdadm, and /dev/disks/by-label/ I should be fine with just swapping the SATA cables around actually, without having to change any configs. I'll try that later on and let you know if it affects performance anything. Good catch! I'm not prepared to mess around with partitions on the RAID drives, as I have data on them I wish to keep. // Mathias -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 12:12 ` Mathias Burén @ 2011-01-30 19:44 ` Stan Hoeppner 2011-01-30 19:46 ` Mathias Burén 2011-02-01 11:37 ` John Robinson 1 sibling, 1 reply; 37+ messages in thread From: Stan Hoeppner @ 2011-01-30 19:44 UTC (permalink / raw) To: Mathias Burén; +Cc: Keld Jørn Simonsen, Linux-RAID Mathias Burén put forth on 1/30/2011 6:12 AM: > Thanks to mdadm, and /dev/disks/by-label/ I should be fine with just > swapping the SATA cables around actually, without having to change any > configs. I'll try that later on and let you know if it affects > performance anything. Good catch! Be sure to change the mobo and PCIe card BIOS boot order after moving the SSD cable to the PCIe card or the machine won't boot. Also, before swapping cables, make sure you have the PCIe card chipset driver built into your kernel or properly built into your initrd image. If not you still won't boot. -- Stan -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 19:44 ` Stan Hoeppner @ 2011-01-30 19:46 ` Mathias Burén 0 siblings, 0 replies; 37+ messages in thread From: Mathias Burén @ 2011-01-30 19:46 UTC (permalink / raw) To: Stan Hoeppner; +Cc: Keld Jørn Simonsen, Linux-RAID On 30 January 2011 19:44, Stan Hoeppner <stan@hardwarefreak.com> wrote: > Mathias Burén put forth on 1/30/2011 6:12 AM: > >> Thanks to mdadm, and /dev/disks/by-label/ I should be fine with just >> swapping the SATA cables around actually, without having to change any >> configs. I'll try that later on and let you know if it affects >> performance anything. Good catch! > > Be sure to change the mobo and PCIe card BIOS boot order after moving the SSD > cable to the PCIe card or the machine won't boot. Also, before swapping cables, > make sure you have the PCIe card chipset driver built into your kernel or > properly built into your initrd image. If not you still won't boot. > > -- > Stan > I forgot to mention it, but I already swapped the cables. I had to change a few things (reinstall GRUB, change menu.lst) but otherwise I was OK. Strange thing is that my mainboard doesn't seem to want to boot off a drive that's connected to the Highpoint controller. Just sits there. However, using GRUB from a USB stick enables me to just do root (hd2,0), then chainload + and load the GRUB of the SSD. Not elegant, but it works (for now). Will investigate more later... // Mathias -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 12:12 ` Mathias Burén 2011-01-30 19:44 ` Stan Hoeppner @ 2011-02-01 11:37 ` John Robinson 2011-02-01 13:53 ` Roberto Spadim 2011-02-01 14:02 ` Mathias Burén 1 sibling, 2 replies; 37+ messages in thread From: John Robinson @ 2011-02-01 11:37 UTC (permalink / raw) To: Mathias Burén; +Cc: Linux-RAID On 30/01/2011 12:12, Mathias Burén wrote: [...] > Ah, good point. The sda is a 60GB SSD, I should definitely move that > to the PCI-E card, as it doesn't do heavy IO (just small random r/w). > Then i can have 4 RAID HDDs on the onboard ctrl, and 2 on the PCI-E > shared with the SSD. If the SSD is idle then I should get ideal > throughputs. Hmm, if you have an SSD, you might look at using bcache to speed up write access to your array. On the other hand, with only one SSD you potentially lose redundancy - do SSDs crash and burn like hard drives do? If the SSD is only being used as a boot/OS drive - so near idle in normal use - I'd swap it for a cheap small laptop hard drive and find somewhere else to put the SSD to better use. Cheers, John. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-02-01 11:37 ` John Robinson @ 2011-02-01 13:53 ` Roberto Spadim 2011-02-01 14:02 ` Mathias Burén 1 sibling, 0 replies; 37+ messages in thread From: Roberto Spadim @ 2011-02-01 13:53 UTC (permalink / raw) To: John Robinson; +Cc: Mathias Burén, Linux-RAID if money isn´t a problem, buy only ssd =) they have ecc, faster read/write, less latency, and a bigger MTBF (they are better!) and they broken too, replace every 5 years to don´t have problems with loose of information 2011/2/1 John Robinson <john.robinson@anonymous.org.uk>: > On 30/01/2011 12:12, Mathias Burén wrote: > [...] >> >> Ah, good point. The sda is a 60GB SSD, I should definitely move that >> to the PCI-E card, as it doesn't do heavy IO (just small random r/w). >> Then i can have 4 RAID HDDs on the onboard ctrl, and 2 on the PCI-E >> shared with the SSD. If the SSD is idle then I should get ideal >> throughputs. > > Hmm, if you have an SSD, you might look at using bcache to speed up write > access to your array. On the other hand, with only one SSD you potentially > lose redundancy - do SSDs crash and burn like hard drives do? > > If the SSD is only being used as a boot/OS drive - so near idle in normal > use - I'd swap it for a cheap small laptop hard drive and find somewhere > else to put the SSD to better use. > > Cheers, > > John. > > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Roberto Spadim Spadim Technology / SPAEmpresarial -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-02-01 11:37 ` John Robinson 2011-02-01 13:53 ` Roberto Spadim @ 2011-02-01 14:02 ` Mathias Burén 2011-02-01 14:32 ` Roberto Spadim 1 sibling, 1 reply; 37+ messages in thread From: Mathias Burén @ 2011-02-01 14:02 UTC (permalink / raw) To: John Robinson; +Cc: Linux-RAID On 1 February 2011 11:37, John Robinson <john.robinson@anonymous.org.uk> wrote: > On 30/01/2011 12:12, Mathias Burén wrote: > [...] >> >> Ah, good point. The sda is a 60GB SSD, I should definitely move that >> to the PCI-E card, as it doesn't do heavy IO (just small random r/w). >> Then i can have 4 RAID HDDs on the onboard ctrl, and 2 on the PCI-E >> shared with the SSD. If the SSD is idle then I should get ideal >> throughputs. > > Hmm, if you have an SSD, you might look at using bcache to speed up write > access to your array. On the other hand, with only one SSD you potentially > lose redundancy - do SSDs crash and burn like hard drives do? > > If the SSD is only being used as a boot/OS drive - so near idle in normal > use - I'd swap it for a cheap small laptop hard drive and find somewhere > else to put the SSD to better use. > > Cheers, > > John. > > Thanks for the input. My initial question was basically, "where's my bottleneck". The SSD is hosting the OS + an XBMC database :-) so I can't use it for anything else. // Mathias -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-02-01 14:02 ` Mathias Burén @ 2011-02-01 14:32 ` Roberto Spadim 0 siblings, 0 replies; 37+ messages in thread From: Roberto Spadim @ 2011-02-01 14:32 UTC (permalink / raw) To: Mathias Burén; +Cc: John Robinson, Linux-RAID nice, with my ocz vertex2 tests i get this: sata1 can get about 130MB/s (sata1 = 1,5gb/s) sata2 can get about 270MB/s (sata1 = 3gb/s) i don´t remember but i think that each ´x´ on pci-express = 2.5gb/s, 4x=10gb/s onboard sata controllers are pci-express or pci based board (it´s hard linked in north bridge) 2011/2/1 Mathias Burén <mathias.buren@gmail.com>: > On 1 February 2011 11:37, John Robinson <john.robinson@anonymous.org.uk> wrote: >> On 30/01/2011 12:12, Mathias Burén wrote: >> [...] >>> >>> Ah, good point. The sda is a 60GB SSD, I should definitely move that >>> to the PCI-E card, as it doesn't do heavy IO (just small random r/w). >>> Then i can have 4 RAID HDDs on the onboard ctrl, and 2 on the PCI-E >>> shared with the SSD. If the SSD is idle then I should get ideal >>> throughputs. >> >> Hmm, if you have an SSD, you might look at using bcache to speed up write >> access to your array. On the other hand, with only one SSD you potentially >> lose redundancy - do SSDs crash and burn like hard drives do? >> >> If the SSD is only being used as a boot/OS drive - so near idle in normal >> use - I'd swap it for a cheap small laptop hard drive and find somewhere >> else to put the SSD to better use. >> >> Cheers, >> >> John. >> >> > > Thanks for the input. My initial question was basically, "where's my > bottleneck". The SSD is hosting the OS + an XBMC database :-) so I > can't use it for anything else. > > // Mathias > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Roberto Spadim Spadim Technology / SPAEmpresarial -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-29 22:48 Performance question, RAID5 Mathias Burén 2011-01-29 22:53 ` Roman Mamedov @ 2011-01-29 23:26 ` CoolCold 2011-01-30 0:18 ` Mathias Burén 1 sibling, 1 reply; 37+ messages in thread From: CoolCold @ 2011-01-29 23:26 UTC (permalink / raw) To: Mathias Burén; +Cc: Linux-RAID You may need to increase stripe cache size http://peterkieser.com/2009/11/29/raid-mdraid-stripe_cache_size-vs-write-transfer/ On Sun, Jan 30, 2011 at 1:48 AM, Mathias Burén <mathias.buren@gmail.com> wrote: > Hi, > > I'm wondering if the performance I'm getting is OK or if there's > something I can do about it. Also, where the potential bottlenecks > are. > > Setup: 6x2TB HDDs, their performance: > > /dev/sdb: > Timing cached reads: 1322 MB in 2.00 seconds = 661.51 MB/sec > Timing buffered disk reads: 362 MB in 3.02 seconds = 120.06 MB/sec > > /dev/sdc: > Timing cached reads: 1282 MB in 2.00 seconds = 641.20 MB/sec > Timing buffered disk reads: 342 MB in 3.01 seconds = 113.53 MB/sec > > /dev/sdd: > Timing cached reads: 1282 MB in 2.00 seconds = 640.55 MB/sec > Timing buffered disk reads: 344 MB in 3.00 seconds = 114.58 MB/sec > > /dev/sde: > Timing cached reads: 1328 MB in 2.00 seconds = 664.46 MB/sec > Timing buffered disk reads: 350 MB in 3.01 seconds = 116.37 MB/sec > > /dev/sdf: > Timing cached reads: 1304 MB in 2.00 seconds = 651.55 MB/sec > Timing buffered disk reads: 378 MB in 3.01 seconds = 125.62 MB/sec > > /dev/sdg: > Timing cached reads: 1324 MB in 2.00 seconds = 661.91 MB/sec > Timing buffered disk reads: 400 MB in 3.00 seconds = 133.15 MB/sec > > These are used in a RAID5 setup: > > Personalities : [raid6] [raid5] [raid4] > md0 : active raid5 sdf1[0] sdg1[6] sde1[5] sdc1[3] sdd1[4] sdb1[1] > 9751756800 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU] > > unused devices: <none> > > /dev/md0: > Version : 1.2 > Creation Time : Tue Oct 19 08:58:41 2010 > Raid Level : raid5 > Array Size : 9751756800 (9300.00 GiB 9985.80 GB) > Used Dev Size : 1950351360 (1860.00 GiB 1997.16 GB) > Raid Devices : 6 > Total Devices : 6 > Persistence : Superblock is persistent > > Update Time : Fri Jan 28 14:55:48 2011 > State : clean > Active Devices : 6 > Working Devices : 6 > Failed Devices : 0 > Spare Devices : 0 > > Layout : left-symmetric > Chunk Size : 64K > > Name : ion:0 (local to host ion) > UUID : e6595c64:b3ae90b3:f01133ac:3f402d20 > Events : 3035769 > > Number Major Minor RaidDevice State > 0 8 81 0 active sync /dev/sdf1 > 1 8 17 1 active sync /dev/sdb1 > 4 8 49 2 active sync /dev/sdd1 > 3 8 33 3 active sync /dev/sdc1 > 5 8 65 4 active sync /dev/sde1 > 6 8 97 5 active sync /dev/sdg1 > > As you can see they are partitioned. They are all identical like this: > > Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes > 81 heads, 63 sectors/track, 765633 cylinders, total 3907029168 sectors > Units = sectors of 1 * 512 = 512 bytes > Sector size (logical/physical): 512 bytes / 512 bytes > I/O size (minimum/optimal): 512 bytes / 512 bytes > Disk identifier: 0x0e5b3a7a > > Device Boot Start End Blocks Id System > /dev/sdb1 2048 3907029167 1953513560 fd Linux raid autodetect > > On this I run LVM: > > --- Physical volume --- > PV Name /dev/md0 > VG Name lvstorage > PV Size 9.08 TiB / not usable 1.00 MiB > Allocatable yes (but full) > PE Size 1.00 MiB > Total PE 9523199 > Free PE 0 > Allocated PE 9523199 > PV UUID YLEUKB-klxF-X3gF-6dG3-DL4R-xebv-6gKQc2 > > On top of the LVM I have: > > --- Volume group --- > VG Name lvstorage > System ID > Format lvm2 > Metadata Areas 1 > Metadata Sequence No 6 > VG Access read/write > VG Status resizable > MAX LV 0 > Cur LV 1 > Open LV 1 > Max PV 0 > Cur PV 1 > Act PV 1 > VG Size 9.08 TiB > PE Size 1.00 MiB > Total PE 9523199 > Alloc PE / Size 9523199 / 9.08 TiB > Free PE / Size 0 / 0 > VG UUID Xd0HTM-azdN-v9kJ-C7vD-COcU-Cnn8-6AJ6hI > > And in turn: > > --- Logical volume --- > LV Name /dev/lvstorage/storage > VG Name lvstorage > LV UUID 9wsJ0u-0QMs-lL5h-E2UA-7QJa-l46j-oWkSr3 > LV Write Access read/write > LV Status available > # open 1 > LV Size 9.08 TiB > Current LE 9523199 > Segments 1 > Allocation inherit > Read ahead sectors auto > - currently set to 1280 > Block device 254:1 > > And on that (sorry) there's the ext4 partition: > > /dev/mapper/lvstorage-storage on /raid5volume type ext4 > (rw,noatime,barrier=1,nouser_xattr) > > Here are the numbers: > > /raid5volume $ time dd if=/dev/zero of=./bigfile.tmp bs=1M count=8192 > 8192+0 records in > 8192+0 records out > 8589934592 bytes (8.6 GB) copied, 94.0967 s, 91.3 MB/s > > real 1m34.102s > user 0m0.107s > sys 0m54.693s > > /raid5volume $ time dd if=./bigfile.tmp of=/dev/null bs=1M > 8192+0 records in > 8192+0 records out > 8589934592 bytes (8.6 GB) copied, 37.8557 s, 227 MB/s > > real 0m37.861s > user 0m0.053s > sys 0m23.608s > > I saw that the process md0_raid5 spike sometimes on CPU usage. This is > an Atom @ 1.6GHz, is that what is limiting the results? Here's > bonnie++: > > /raid5volume/temp $ time bonnie++ -d ./ -m ion > Writing with putc()...done > Writing intelligently...done > Rewriting...done > Reading with getc()...done > Reading intelligently...done > start 'em...done...done...done... > Create files in sequential order...done. > Stat files in sequential order...done. > Delete files in sequential order...done. > Create files in random order...done. > Stat files in random order...done. > Delete files in random order...done. > Version 1.03e ------Sequential Output------ --Sequential Input- --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP > ion 7G 13726 98 148051 87 68020 41 14547 99 286647 > 61 404.1 2 > ------Sequential Create------ --------Random Create-------- > -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > 16 20707 99 +++++ +++ 25870 99 21242 98 +++++ +++ 25630 100 > ion,7G,13726,98,148051,87,68020,41,14547,99,286647,61,404.1,2,16,20707,99,+++++,+++,25870,99,21242,98,+++++,+++,25630,100 > > real 20m54.320s > user 16m10.447s > sys 2m45.543s > > > Thanks in advance, > // Mathias > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Best regards, [COOLCOLD-RIPN] -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-29 23:26 ` CoolCold @ 2011-01-30 0:18 ` Mathias Burén 2011-01-30 4:44 ` Roman Mamedov 0 siblings, 1 reply; 37+ messages in thread From: Mathias Burén @ 2011-01-30 0:18 UTC (permalink / raw) To: CoolCold; +Cc: Linux-RAID On 29 January 2011 23:26, CoolCold <coolthecold@gmail.com> wrote: > You may need to increase stripe cache size > http://peterkieser.com/2009/11/29/raid-mdraid-stripe_cache_size-vs-write-transfer/ > > On Sun, Jan 30, 2011 at 1:48 AM, Mathias Burén <mathias.buren@gmail.com> wrote: >> Hi, >> >> I'm wondering if the performance I'm getting is OK or if there's >> something I can do about it. Also, where the potential bottlenecks >> are. >> >> Setup: 6x2TB HDDs, their performance: >> >> /dev/sdb: >> Timing cached reads: 1322 MB in 2.00 seconds = 661.51 MB/sec >> Timing buffered disk reads: 362 MB in 3.02 seconds = 120.06 MB/sec >> >> /dev/sdc: >> Timing cached reads: 1282 MB in 2.00 seconds = 641.20 MB/sec >> Timing buffered disk reads: 342 MB in 3.01 seconds = 113.53 MB/sec >> >> /dev/sdd: >> Timing cached reads: 1282 MB in 2.00 seconds = 640.55 MB/sec >> Timing buffered disk reads: 344 MB in 3.00 seconds = 114.58 MB/sec >> >> /dev/sde: >> Timing cached reads: 1328 MB in 2.00 seconds = 664.46 MB/sec >> Timing buffered disk reads: 350 MB in 3.01 seconds = 116.37 MB/sec >> >> /dev/sdf: >> Timing cached reads: 1304 MB in 2.00 seconds = 651.55 MB/sec >> Timing buffered disk reads: 378 MB in 3.01 seconds = 125.62 MB/sec >> >> /dev/sdg: >> Timing cached reads: 1324 MB in 2.00 seconds = 661.91 MB/sec >> Timing buffered disk reads: 400 MB in 3.00 seconds = 133.15 MB/sec >> >> These are used in a RAID5 setup: >> >> Personalities : [raid6] [raid5] [raid4] >> md0 : active raid5 sdf1[0] sdg1[6] sde1[5] sdc1[3] sdd1[4] sdb1[1] >> 9751756800 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU] >> >> unused devices: <none> >> >> /dev/md0: >> Version : 1.2 >> Creation Time : Tue Oct 19 08:58:41 2010 >> Raid Level : raid5 >> Array Size : 9751756800 (9300.00 GiB 9985.80 GB) >> Used Dev Size : 1950351360 (1860.00 GiB 1997.16 GB) >> Raid Devices : 6 >> Total Devices : 6 >> Persistence : Superblock is persistent >> >> Update Time : Fri Jan 28 14:55:48 2011 >> State : clean >> Active Devices : 6 >> Working Devices : 6 >> Failed Devices : 0 >> Spare Devices : 0 >> >> Layout : left-symmetric >> Chunk Size : 64K >> >> Name : ion:0 (local to host ion) >> UUID : e6595c64:b3ae90b3:f01133ac:3f402d20 >> Events : 3035769 >> >> Number Major Minor RaidDevice State >> 0 8 81 0 active sync /dev/sdf1 >> 1 8 17 1 active sync /dev/sdb1 >> 4 8 49 2 active sync /dev/sdd1 >> 3 8 33 3 active sync /dev/sdc1 >> 5 8 65 4 active sync /dev/sde1 >> 6 8 97 5 active sync /dev/sdg1 >> >> As you can see they are partitioned. They are all identical like this: >> >> Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes >> 81 heads, 63 sectors/track, 765633 cylinders, total 3907029168 sectors >> Units = sectors of 1 * 512 = 512 bytes >> Sector size (logical/physical): 512 bytes / 512 bytes >> I/O size (minimum/optimal): 512 bytes / 512 bytes >> Disk identifier: 0x0e5b3a7a >> >> Device Boot Start End Blocks Id System >> /dev/sdb1 2048 3907029167 1953513560 fd Linux raid autodetect >> >> On this I run LVM: >> >> --- Physical volume --- >> PV Name /dev/md0 >> VG Name lvstorage >> PV Size 9.08 TiB / not usable 1.00 MiB >> Allocatable yes (but full) >> PE Size 1.00 MiB >> Total PE 9523199 >> Free PE 0 >> Allocated PE 9523199 >> PV UUID YLEUKB-klxF-X3gF-6dG3-DL4R-xebv-6gKQc2 >> >> On top of the LVM I have: >> >> --- Volume group --- >> VG Name lvstorage >> System ID >> Format lvm2 >> Metadata Areas 1 >> Metadata Sequence No 6 >> VG Access read/write >> VG Status resizable >> MAX LV 0 >> Cur LV 1 >> Open LV 1 >> Max PV 0 >> Cur PV 1 >> Act PV 1 >> VG Size 9.08 TiB >> PE Size 1.00 MiB >> Total PE 9523199 >> Alloc PE / Size 9523199 / 9.08 TiB >> Free PE / Size 0 / 0 >> VG UUID Xd0HTM-azdN-v9kJ-C7vD-COcU-Cnn8-6AJ6hI >> >> And in turn: >> >> --- Logical volume --- >> LV Name /dev/lvstorage/storage >> VG Name lvstorage >> LV UUID 9wsJ0u-0QMs-lL5h-E2UA-7QJa-l46j-oWkSr3 >> LV Write Access read/write >> LV Status available >> # open 1 >> LV Size 9.08 TiB >> Current LE 9523199 >> Segments 1 >> Allocation inherit >> Read ahead sectors auto >> - currently set to 1280 >> Block device 254:1 >> >> And on that (sorry) there's the ext4 partition: >> >> /dev/mapper/lvstorage-storage on /raid5volume type ext4 >> (rw,noatime,barrier=1,nouser_xattr) >> >> Here are the numbers: >> >> /raid5volume $ time dd if=/dev/zero of=./bigfile.tmp bs=1M count=8192 >> 8192+0 records in >> 8192+0 records out >> 8589934592 bytes (8.6 GB) copied, 94.0967 s, 91.3 MB/s >> >> real 1m34.102s >> user 0m0.107s >> sys 0m54.693s >> >> /raid5volume $ time dd if=./bigfile.tmp of=/dev/null bs=1M >> 8192+0 records in >> 8192+0 records out >> 8589934592 bytes (8.6 GB) copied, 37.8557 s, 227 MB/s >> >> real 0m37.861s >> user 0m0.053s >> sys 0m23.608s >> >> I saw that the process md0_raid5 spike sometimes on CPU usage. This is >> an Atom @ 1.6GHz, is that what is limiting the results? Here's >> bonnie++: >> >> /raid5volume/temp $ time bonnie++ -d ./ -m ion >> Writing with putc()...done >> Writing intelligently...done >> Rewriting...done >> Reading with getc()...done >> Reading intelligently...done >> start 'em...done...done...done... >> Create files in sequential order...done. >> Stat files in sequential order...done. >> Delete files in sequential order...done. >> Create files in random order...done. >> Stat files in random order...done. >> Delete files in random order...done. >> Version 1.03e ------Sequential Output------ --Sequential Input- --Random- >> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- >> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP >> ion 7G 13726 98 148051 87 68020 41 14547 99 286647 >> 61 404.1 2 >> ------Sequential Create------ --------Random Create-------- >> -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- >> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> 16 20707 99 +++++ +++ 25870 99 21242 98 +++++ +++ 25630 100 >> ion,7G,13726,98,148051,87,68020,41,14547,99,286647,61,404.1,2,16,20707,99,+++++,+++,25870,99,21242,98,+++++,+++,25630,100 >> >> real 20m54.320s >> user 16m10.447s >> sys 2m45.543s >> >> >> Thanks in advance, >> // Mathias >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > > > > -- > Best regards, > [COOLCOLD-RIPN] > I ran the benchmark found on the page (except for writes); results: tripe_cache_size: 256 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 63.4351 s, 271 MB/s stripe_cache_size: 256 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 59.8224 s, 287 MB/s stripe_cache_size: 256 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 62.1066 s, 277 MB/s stripe_cache_size: 512 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 59.6833 s, 288 MB/s stripe_cache_size: 512 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 60.3497 s, 285 MB/s stripe_cache_size: 512 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 59.7853 s, 287 MB/s stripe_cache_size: 768 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 59.5398 s, 288 MB/s stripe_cache_size: 768 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 60.1769 s, 285 MB/s stripe_cache_size: 768 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 60.5354 s, 284 MB/s stripe_cache_size: 1024 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 60.1814 s, 285 MB/s stripe_cache_size: 1024 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.6288 s, 279 MB/s stripe_cache_size: 1024 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.9942 s, 277 MB/s stripe_cache_size: 2048 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.177 s, 281 MB/s stripe_cache_size: 2048 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.3905 s, 280 MB/s stripe_cache_size: 2048 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.0274 s, 281 MB/s stripe_cache_size: 4096 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 62.607 s, 274 MB/s stripe_cache_size: 4096 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 63.1505 s, 272 MB/s stripe_cache_size: 4096 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.4747 s, 279 MB/s stripe_cache_size: 8192 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 62.0839 s, 277 MB/s stripe_cache_size: 8192 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 62.7944 s, 274 MB/s stripe_cache_size: 8192 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.4443 s, 280 MB/s stripe_cache_size: 16834 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.9554 s, 277 MB/s stripe_cache_size: 16834 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 63.8002 s, 269 MB/s stripe_cache_size: 16834 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 62.2772 s, 276 MB/s stripe_cache_size: 32768 (1/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 62.4692 s, 275 MB/s stripe_cache_size: 32768 (2/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 61.6707 s, 279 MB/s stripe_cache_size: 32768 (3/3) 5460+0 records in 5460+0 records out 17175674880 bytes (17 GB) copied, 63.4744 s, 271 MB/s It looks like a small stripe cache is favoured here. // Mathias -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 0:18 ` Mathias Burén @ 2011-01-30 4:44 ` Roman Mamedov 2011-01-30 12:09 ` Mathias Burén 0 siblings, 1 reply; 37+ messages in thread From: Roman Mamedov @ 2011-01-30 4:44 UTC (permalink / raw) To: Mathias Burén; +Cc: CoolCold, Linux-RAID [-- Attachment #1: Type: text/plain, Size: 270 bytes --] On Sun, 30 Jan 2011 00:18:34 +0000 Mathias Burén <mathias.buren@gmail.com> wrote: > I ran the benchmark found on the page (except for writes); results: That's kinda unfortunate, as stripe_cache_size only (or mostly) affects writes. -- With respect, Roman [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 4:44 ` Roman Mamedov @ 2011-01-30 12:09 ` Mathias Burén 2011-01-30 12:15 ` Roman Mamedov 0 siblings, 1 reply; 37+ messages in thread From: Mathias Burén @ 2011-01-30 12:09 UTC (permalink / raw) To: Roman Mamedov; +Cc: CoolCold, Linux-RAID On 30 January 2011 04:44, Roman Mamedov <rm@romanrm.ru> wrote: > On Sun, 30 Jan 2011 00:18:34 +0000 > Mathias Burén <mathias.buren@gmail.com> wrote: > >> I ran the benchmark found on the page (except for writes); results: > > That's kinda unfortunate, as stripe_cache_size only (or mostly) affects writes. > > -- > With respect, > Roman > Right, it's just that I don't want to destroy my data. I've ran a few bonnie++ benchmarks with different mount options though. You can find them here: http://stuff.dyndns.org/logs/bonnie_results.html It actually looks like stripe=384 helped performance a bit. Currently retrying the same mount options but with 32MB stripe cache instead of 8MB. Then you have all the readahead settings as well, like: blockdev --setra 8192 /dev/sd[abcdefgh] blockdev --setra 65536 /dev/md0 And disabling NCQ; for i in sdb sdc sdd sde sdf sdg; do echo 1 > /sys/block/"$i"/device/queue_depth; done I'll try those settings later. // Mathias -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 12:09 ` Mathias Burén @ 2011-01-30 12:15 ` Roman Mamedov 2011-01-30 19:41 ` Mathias Burén 0 siblings, 1 reply; 37+ messages in thread From: Roman Mamedov @ 2011-01-30 12:15 UTC (permalink / raw) To: Mathias Burén; +Cc: CoolCold, Linux-RAID [-- Attachment #1: Type: text/plain, Size: 668 bytes --] On Sun, 30 Jan 2011 12:09:02 +0000 Mathias Burén <mathias.buren@gmail.com> wrote: > Right, it's just that I don't want to destroy my data. I've ran a few > bonnie++ benchmarks with different mount options though. You can find > them here: http://stuff.dyndns.org/logs/bonnie_results.html > It actually looks like stripe=384 helped performance a bit. Currently > retrying the same mount options but with 32MB stripe cache instead of > 8MB. Be aware that it's not just 32MB of RAM, it's "stripe_cache_size * 4096 (page size) * number of disks". In other words on 6 disks this stripe cache will consume 768 MB of RAM. -- With respect, Roman [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 12:15 ` Roman Mamedov @ 2011-01-30 19:41 ` Mathias Burén 2011-01-30 19:54 ` Roman Mamedov 2011-01-30 20:03 ` Stan Hoeppner 0 siblings, 2 replies; 37+ messages in thread From: Mathias Burén @ 2011-01-30 19:41 UTC (permalink / raw) To: Roman Mamedov; +Cc: CoolCold, Linux-RAID On 30 January 2011 12:15, Roman Mamedov <rm@romanrm.ru> wrote: > On Sun, 30 Jan 2011 12:09:02 +0000 > Mathias Burén <mathias.buren@gmail.com> wrote: > >> Right, it's just that I don't want to destroy my data. I've ran a few >> bonnie++ benchmarks with different mount options though. You can find >> them here: http://stuff.dyndns.org/logs/bonnie_results.html >> It actually looks like stripe=384 helped performance a bit. Currently >> retrying the same mount options but with 32MB stripe cache instead of >> 8MB. > > Be aware that it's not just 32MB of RAM, it's > > "stripe_cache_size * 4096 (page size) * number of disks". > > In other words on 6 disks this stripe cache will consume 768 MB of RAM. > > -- > With respect, > Roman > Thanks. New results up for those interested: http://stuff.dyndns.org/logs/bonnie_results.html It looks like the best performance (for me) is gained using rw,noatime,nouser_xattr,data=writeback,stripe=384 , 8192 stripe_cache_size, NCQ turned on (31), md0 readahead of 65536. I did switch place between the SSD and a HDD so there's only 2 HDDs on the PCI-E controller now, that are part of the RAID. The other 4 are on the internal SATA controller. csv format: "Host","Chunk size","Sequential Output",,,,,,"Sequential Input",,,,"Random Seeks",,"Files","Sequential Create",,,,,,"Random Create",,,,,,"Mount options","stripe_cache_size","NCQ","md0 readahead","Comment" ,,"Per Chr",,"Block",,"Rewrite",,"Per Chr",,"Block",,"Seeks",,,"Create",,"Read",,"Delete",,"Create",,"Read",,"Delete",,,,,, ,,"K/sec","CPU","K/sec","CPU","K/sec","CPU","K/sec","CPU","K/sec","CPU","/sec","CPU",,"/sec","CPU","/sec","CPU","/sec","CPU","/sec","CPU","/sec","CPU","/sec","CPU",,,,, "ion","7G",13916,98,158905,91,71718,39,14539,99,295079,57,482.8,3,16,20747,99,"+++++","+++",24077,93,21249,99,"+++++","+++",25854,99,"rw,noatime,nouser_xattr,data=writeback,stripe=384",8192,31,65536,"4 on host 2 on pci-e" A bit messy. I did find another PCI-E SATA controller that is generation 2.0, and looks like it may do the trick. This one: http://www.newegg.com/Product/Product.aspx?Item=N82E16816115053 "HighPoint RocketRAID 2640x1 PCI-Express x1 Four-Port SATA and SAS RAID Controller Card". It's ~€110 on eBay, a bit hefty. I might just save up and build a NAS box from scratch, with some mainboard which has 8 SATA from the start etc. // Mathias -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 19:41 ` Mathias Burén @ 2011-01-30 19:54 ` Roman Mamedov 2011-01-30 19:58 ` Mathias Burén 2011-01-30 20:03 ` Stan Hoeppner 1 sibling, 1 reply; 37+ messages in thread From: Roman Mamedov @ 2011-01-30 19:54 UTC (permalink / raw) To: Mathias Burén; +Cc: CoolCold, Linux-RAID [-- Attachment #1: Type: text/plain, Size: 760 bytes --] On Sun, 30 Jan 2011 19:41:47 +0000 Mathias Burén <mathias.buren@gmail.com> wrote: > I did find another PCI-E SATA controller that is generation 2.0, and > looks like it may do the trick. This one: > http://www.newegg.com/Product/Product.aspx?Item=N82E16816115053 > "HighPoint RocketRAID 2640x1 PCI-Express x1 Four-Port SATA and SAS > RAID Controller Card". > > It's ~€110 on eBay, a bit hefty. I might just save up and build a NAS > box from scratch, with some mainboard which has 8 SATA from the start > etc. This one is cheaper: http://www.dealextreme.com/p/lsi-sas3041e-r-4-port-sas-sata-host-bus-adapter-51317 Doesn't matter if it's v1.x or v2.0, since it's x4, it will have enough bandwidth either way. -- With respect, Roman [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 19:54 ` Roman Mamedov @ 2011-01-30 19:58 ` Mathias Burén 0 siblings, 0 replies; 37+ messages in thread From: Mathias Burén @ 2011-01-30 19:58 UTC (permalink / raw) To: Roman Mamedov; +Cc: CoolCold, Linux-RAID On 30 January 2011 19:54, Roman Mamedov <rm@romanrm.ru> wrote: > On Sun, 30 Jan 2011 19:41:47 +0000 > Mathias Burén <mathias.buren@gmail.com> wrote: > >> I did find another PCI-E SATA controller that is generation 2.0, and >> looks like it may do the trick. This one: >> http://www.newegg.com/Product/Product.aspx?Item=N82E16816115053 >> "HighPoint RocketRAID 2640x1 PCI-Express x1 Four-Port SATA and SAS >> RAID Controller Card". >> >> It's ~€110 on eBay, a bit hefty. I might just save up and build a NAS >> box from scratch, with some mainboard which has 8 SATA from the start >> etc. > > This one is cheaper: > http://www.dealextreme.com/p/lsi-sas3041e-r-4-port-sas-sata-host-bus-adapter-51317 > Doesn't matter if it's v1.x or v2.0, since it's x4, it will have enough > bandwidth either way. > > -- > With respect, > Roman > Sorry, my current mainboard (it's a Zotac Ion/Atom ITX) only has one x1 connector (2.0), hence my limited choice of cards. // Mathias -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 19:41 ` Mathias Burén 2011-01-30 19:54 ` Roman Mamedov @ 2011-01-30 20:03 ` Stan Hoeppner 2011-01-30 21:43 ` Mathias Burén 1 sibling, 1 reply; 37+ messages in thread From: Stan Hoeppner @ 2011-01-30 20:03 UTC (permalink / raw) To: Mathias Burén; +Cc: Roman Mamedov, CoolCold, Linux-RAID Mathias Burén put forth on 1/30/2011 1:41 PM: > Thanks. New results up for those interested: > http://stuff.dyndns.org/logs/bonnie_results.html Three things I notice: 1. You're CPU bound across the board, for all the tests that matter anyway 2. Because of this, your performance spread is less than 5% across the board meaning any optimizations are useless 3. Is that a 7 Gigabyte chunk size? That's totally unrealistic. It should be less than 1 MB for almost all workloads. According to those numbers, you can swap SATA controllers and PCIe bus slot assignments all day long, but you'll gain nothing without a faster CPU. Why are you using a 7 Gigabyte chunk size? And if the other OP was correct about the 768MB stripe cache, that's totally unrealistic as well. And in real world use, you don't want a high readahead setting. It just wastes buffer cache memory for no gain (except maybe in some synthetic benchmarks). -- Stan -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 20:03 ` Stan Hoeppner @ 2011-01-30 21:43 ` Mathias Burén 2011-01-31 3:39 ` Stan Hoeppner 2011-01-31 8:52 ` Keld Jørn Simonsen 0 siblings, 2 replies; 37+ messages in thread From: Mathias Burén @ 2011-01-30 21:43 UTC (permalink / raw) To: Stan Hoeppner; +Cc: Roman Mamedov, CoolCold, Linux-RAID On 30 January 2011 20:03, Stan Hoeppner <stan@hardwarefreak.com> wrote: > Mathias Burén put forth on 1/30/2011 1:41 PM: > >> Thanks. New results up for those interested: >> http://stuff.dyndns.org/logs/bonnie_results.html > > Three things I notice: > > 1. You're CPU bound across the board, for all the tests that matter anyway > 2. Because of this, your performance spread is less than 5% across the board > meaning any optimizations are useless > 3. Is that a 7 Gigabyte chunk size? That's totally unrealistic. It should be > less than 1 MB for almost all workloads. > > According to those numbers, you can swap SATA controllers and PCIe bus slot > assignments all day long, but you'll gain nothing without a faster CPU. > > Why are you using a 7 Gigabyte chunk size? And if the other OP was correct > about the 768MB stripe cache, that's totally unrealistic as well. And in real > world use, you don't want a high readahead setting. It just wastes buffer cache > memory for no gain (except maybe in some synthetic benchmarks). > > -- > Stan > CPU bound, got it. 3) 7GB output from bonnie++. Like so: $ time bonnie++ -m ion -d ./ Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03e ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP ion 7G 13455 96 150827 92 67719 43 14439 99 271832 59 469.6 3 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 20318 99 +++++ +++ 25428 99 21095 99 +++++ +++ 25278 99 ion,7G,13455,96,150827,92,67719,43,14439,99,271832,59,469.6,3,16,20318,99,+++++,+++,25428,99,21095,99,+++++,+++,25278,99 real 21m7.872s user 16m17.836s sys 2m51.405s (after ion) // Mathias -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 21:43 ` Mathias Burén @ 2011-01-31 3:39 ` Stan Hoeppner 2011-01-31 3:54 ` Roberto Spadim 2011-01-31 8:52 ` Keld Jørn Simonsen 1 sibling, 1 reply; 37+ messages in thread From: Stan Hoeppner @ 2011-01-31 3:39 UTC (permalink / raw) To: Mathias Burén; +Cc: Roman Mamedov, CoolCold, Linux-RAID Mathias Burén put forth on 1/30/2011 3:43 PM: > 3) 7GB output from bonnie++. Like so: The column in your graph http://stuff.dyndns.org/logs/bonnie_results.html says "Chunk size". I'm not a bonnie++ user so my apologies for not being familiar with it. I use iozone instead (on rare occasion) which I like much better. -- Stan -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-31 3:39 ` Stan Hoeppner @ 2011-01-31 3:54 ` Roberto Spadim 0 siblings, 0 replies; 37+ messages in thread From: Roberto Spadim @ 2011-01-31 3:54 UTC (permalink / raw) To: Stan Hoeppner; +Cc: Mathias Burén, Roman Mamedov, CoolCold, Linux-RAID try stress program, and sysstat it make a nice stress test over filesystem (maybe you will use it, only if using database over partition or something familiar) 2011/1/31 Stan Hoeppner <stan@hardwarefreak.com>: > Mathias Burén put forth on 1/30/2011 3:43 PM: > >> 3) 7GB output from bonnie++. Like so: > > The column in your graph http://stuff.dyndns.org/logs/bonnie_results.html says > "Chunk size". I'm not a bonnie++ user so my apologies for not being familiar > with it. I use iozone instead (on rare occasion) which I like much better. > > -- > Stan > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Roberto Spadim Spadim Technology / SPAEmpresarial -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-30 21:43 ` Mathias Burén 2011-01-31 3:39 ` Stan Hoeppner @ 2011-01-31 8:52 ` Keld Jørn Simonsen 2011-01-31 9:37 ` Mathias Burén 1 sibling, 1 reply; 37+ messages in thread From: Keld Jørn Simonsen @ 2011-01-31 8:52 UTC (permalink / raw) To: Mathias Burén; +Cc: Stan Hoeppner, Roman Mamedov, CoolCold, Linux-RAID If your intallation is CPU bound, and you are using an Atom N270 processor or the like, well some ideas: The Atom CPU may have threading, so you could run 2 RAIDs which then probably would run in each thread. It would cost you 1 more disk if you run 2 RAID5's so you get 8 TB payload out of your 12 GB total (6 drives of 2 TB each). Another way to get better performance could be to use less CPU-intensitive RAID types. RAID5 is intensitive as it needs to calculate XOR information all the time. Maybe a mirrored raid type like RAID10,f2 would give you less CPU usage, and the run 2 RAIDS to have it running in both hyperthreads. Here you would then only get 6 TB payload of your 12 GB disks, but then also probably a faster system. Best regards keld ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-31 8:52 ` Keld Jørn Simonsen @ 2011-01-31 9:37 ` Mathias Burén 2011-01-31 13:11 ` Keld Jørn Simonsen 0 siblings, 1 reply; 37+ messages in thread From: Mathias Burén @ 2011-01-31 9:37 UTC (permalink / raw) To: Keld Jørn Simonsen Cc: Stan Hoeppner, Roman Mamedov, CoolCold, Linux-RAID On 31 January 2011 08:52, Keld Jørn Simonsen <keld@keldix.com> wrote: > If your intallation is CPU bound, and you are > using an Atom N270 processor or the like, well some ideas: > > The Atom CPU may have threading, so you could run 2 RAIDs > which then probably would run in each thread. > It would cost you 1 more disk if you run 2 RAID5's > so you get 8 TB payload out of your 12 GB total (6 drives of 2 TB each). > > Another way to get better performance could be to use less > CPU-intensitive RAID types. RAID5 is intensitive as it needs to > calculate XOR information all the time. Maybe a mirrored > raid type like RAID10,f2 would give you less CPU usage, > and the run 2 RAIDS to have it running in both hyperthreads. > Here you would then only get 6 TB payload of your 12 GB disks, > but then also probably a faster system. > > Best regards > keld > Hi, It's interesting what you say about the XOR calculations. I thought that it was only calculated on writes? The Atom (330) has HT, so Linux sees 4 logical CPUs. // Mathias -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-31 9:37 ` Mathias Burén @ 2011-01-31 13:11 ` Keld Jørn Simonsen 2011-01-31 14:43 ` Roberto Spadim 2011-01-31 20:42 ` NeilBrown 0 siblings, 2 replies; 37+ messages in thread From: Keld Jørn Simonsen @ 2011-01-31 13:11 UTC (permalink / raw) To: Mathias Burén Cc: Keld Jørn Simonsen, Stan Hoeppner, Roman Mamedov, CoolCold, Linux-RAID On Mon, Jan 31, 2011 at 09:37:46AM +0000, Mathias Burén wrote: > On 31 January 2011 08:52, Keld Jørn Simonsen <keld@keldix.com> wrote: > > If your intallation is CPU bound, and you are > > using an Atom N270 processor or the like, well some ideas: > > > > The Atom CPU may have threading, so you could run 2 RAIDs > > which then probably would run in each thread. > > It would cost you 1 more disk if you run 2 RAID5's > > so you get 8 TB payload out of your 12 GB total (6 drives of 2 TB each). > > > > Another way to get better performance could be to use less > > CPU-intensitive RAID types. RAID5 is intensitive as it needs to > > calculate XOR information all the time. Maybe a mirrored > > raid type like RAID10,f2 would give you less CPU usage, > > and the run 2 RAIDS to have it running in both hyperthreads. > > Here you would then only get 6 TB payload of your 12 GB disks, > > but then also probably a faster system. > > > > Best regards > > keld > > > > Hi, > > It's interesting what you say about the XOR calculations. I thought > that it was only calculated on writes? The Atom (330) has HT, so Linux > sees 4 logical CPUs. Yes you are right, it only calculates XOR on writes with RAID5. But then I am puzzled what all these CPU cycles are used for. Also many cycles are used on mirrored raid types. Why? Maybe some is because of LVM? I have been puzzled for a long time why ordinary RAID without LVM need to use so much CPU. Maybe a lot of data sguffling between buffers? Neil? Best regards Keld -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-31 13:11 ` Keld Jørn Simonsen @ 2011-01-31 14:43 ` Roberto Spadim 2011-01-31 18:44 ` Keld Jørn Simonsen 2011-01-31 20:42 ` NeilBrown 1 sibling, 1 reply; 37+ messages in thread From: Roberto Spadim @ 2011-01-31 14:43 UTC (permalink / raw) To: Keld Jørn Simonsen Cc: Mathias Burén, Stan Hoeppner, Roman Mamedov, CoolCold, Linux-RAID i think it's cpu wait i/o 2011/1/31 Keld Jørn Simonsen <keld@keldix.com>: > On Mon, Jan 31, 2011 at 09:37:46AM +0000, Mathias Burén wrote: >> On 31 January 2011 08:52, Keld Jørn Simonsen <keld@keldix.com> wrote: >> > If your intallation is CPU bound, and you are >> > using an Atom N270 processor or the like, well some ideas: >> > >> > The Atom CPU may have threading, so you could run 2 RAIDs >> > which then probably would run in each thread. >> > It would cost you 1 more disk if you run 2 RAID5's >> > so you get 8 TB payload out of your 12 GB total (6 drives of 2 TB each). >> > >> > Another way to get better performance could be to use less >> > CPU-intensitive RAID types. RAID5 is intensitive as it needs to >> > calculate XOR information all the time. Maybe a mirrored >> > raid type like RAID10,f2 would give you less CPU usage, >> > and the run 2 RAIDS to have it running in both hyperthreads. >> > Here you would then only get 6 TB payload of your 12 GB disks, >> > but then also probably a faster system. >> > >> > Best regards >> > keld >> > >> >> Hi, >> >> It's interesting what you say about the XOR calculations. I thought >> that it was only calculated on writes? The Atom (330) has HT, so Linux >> sees 4 logical CPUs. > > Yes you are right, it only calculates XOR on writes with RAID5. > But then I am puzzled what all these CPU cycles are used for. > Also many cycles are used on mirrored raid types. Why? > Maybe some is because of LVM? I have been puzzled for a long time why > ordinary RAID without LVM need to use so much CPU. Maybe a lot of data > sguffling between buffers? Neil? > > Best regards > Keld > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Roberto Spadim Spadim Technology / SPAEmpresarial -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-31 14:43 ` Roberto Spadim @ 2011-01-31 18:44 ` Keld Jørn Simonsen 0 siblings, 0 replies; 37+ messages in thread From: Keld Jørn Simonsen @ 2011-01-31 18:44 UTC (permalink / raw) To: Roberto Spadim Cc: Keld Jørn Simonsen, Mathias Burén, Stan Hoeppner, Roman Mamedov, CoolCold, Linux-RAID On Mon, Jan 31, 2011 at 12:43:56PM -0200, Roberto Spadim wrote: > i think it's cpu wait i/o Well, you better be sure. Please have a closer look. Normally saying that a process is CPU bound means that it is not IO bound. Best regards keld > 2011/1/31 Keld Jørn Simonsen <keld@keldix.com>: > > On Mon, Jan 31, 2011 at 09:37:46AM +0000, Mathias Burén wrote: > >> On 31 January 2011 08:52, Keld Jørn Simonsen <keld@keldix.com> wrote: > >> > If your intallation is CPU bound, and you are > >> > using an Atom N270 processor or the like, well some ideas: > >> > > >> > The Atom CPU may have threading, so you could run 2 RAIDs > >> > which then probably would run in each thread. > >> > It would cost you 1 more disk if you run 2 RAID5's > >> > so you get 8 TB payload out of your 12 GB total (6 drives of 2 TB each). > >> > > >> > Another way to get better performance could be to use less > >> > CPU-intensitive RAID types. RAID5 is intensitive as it needs to > >> > calculate XOR information all the time. Maybe a mirrored > >> > raid type like RAID10,f2 would give you less CPU usage, > >> > and the run 2 RAIDS to have it running in both hyperthreads. > >> > Here you would then only get 6 TB payload of your 12 GB disks, > >> > but then also probably a faster system. > >> > > >> > Best regards > >> > keld > >> > > >> > >> Hi, > >> > >> It's interesting what you say about the XOR calculations. I thought > >> that it was only calculated on writes? The Atom (330) has HT, so Linux > >> sees 4 logical CPUs. > > > > Yes you are right, it only calculates XOR on writes with RAID5. > > But then I am puzzled what all these CPU cycles are used for. > > Also many cycles are used on mirrored raid types. Why? > > Maybe some is because of LVM? I have been puzzled for a long time why > > ordinary RAID without LVM need to use so much CPU. Maybe a lot of data > > sguffling between buffers? Neil? > > > > Best regards > > Keld > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > -- > Roberto Spadim > Spadim Technology / SPAEmpresarial -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-31 13:11 ` Keld Jørn Simonsen 2011-01-31 14:43 ` Roberto Spadim @ 2011-01-31 20:42 ` NeilBrown 2011-01-31 21:41 ` Keld Jørn Simonsen 1 sibling, 1 reply; 37+ messages in thread From: NeilBrown @ 2011-01-31 20:42 UTC (permalink / raw) To: Keld Jørn Simonsen Cc: Mathias Burén, Stan Hoeppner, Roman Mamedov, CoolCold, Linux-RAID On Mon, 31 Jan 2011 14:11:31 +0100 Keld Jørn Simonsen <keld@keldix.com> wrote: > On Mon, Jan 31, 2011 at 09:37:46AM +0000, Mathias Burén wrote: > > On 31 January 2011 08:52, Keld Jørn Simonsen <keld@keldix.com> wrote: > > > If your intallation is CPU bound, and you are > > > using an Atom N270 processor or the like, well some ideas: > > > > > > The Atom CPU may have threading, so you could run 2 RAIDs > > > which then probably would run in each thread. > > > It would cost you 1 more disk if you run 2 RAID5's > > > so you get 8 TB payload out of your 12 GB total (6 drives of 2 TB each). > > > > > > Another way to get better performance could be to use less > > > CPU-intensitive RAID types. RAID5 is intensitive as it needs to > > > calculate XOR information all the time. Maybe a mirrored > > > raid type like RAID10,f2 would give you less CPU usage, > > > and the run 2 RAIDS to have it running in both hyperthreads. > > > Here you would then only get 6 TB payload of your 12 GB disks, > > > but then also probably a faster system. > > > > > > Best regards > > > keld > > > > > > > Hi, > > > > It's interesting what you say about the XOR calculations. I thought > > that it was only calculated on writes? The Atom (330) has HT, so Linux > > sees 4 logical CPUs. > > Yes you are right, it only calculates XOR on writes with RAID5. > But then I am puzzled what all these CPU cycles are used for. > Also many cycles are used on mirrored raid types. Why? > Maybe some is because of LVM? I have been puzzled for a long time why > ordinary RAID without LVM need to use so much CPU. Maybe a lot of data > sguffling between buffers? Neil? What is your evidence that RAID1 uses lots of CPU? I would expect it to use very little, but I've been wrong before. NeilBrown -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-31 20:42 ` NeilBrown @ 2011-01-31 21:41 ` Keld Jørn Simonsen 2011-01-31 21:43 ` Roberto Spadim 0 siblings, 1 reply; 37+ messages in thread From: Keld Jørn Simonsen @ 2011-01-31 21:41 UTC (permalink / raw) To: NeilBrown Cc: Keld Jørn Simonsen, Mathias Burén, Stan Hoeppner, Roman Mamedov, CoolCold, Linux-RAID On Tue, Feb 01, 2011 at 07:42:57AM +1100, NeilBrown wrote: > On Mon, 31 Jan 2011 14:11:31 +0100 Keld Jørn Simonsen <keld@keldix.com> wrote: > > > On Mon, Jan 31, 2011 at 09:37:46AM +0000, Mathias Burén wrote: > > > On 31 January 2011 08:52, Keld Jørn Simonsen <keld@keldix.com> wrote: > > > > If your intallation is CPU bound, and you are > > > > using an Atom N270 processor or the like, well some ideas: > > > > > > > > The Atom CPU may have threading, so you could run 2 RAIDs > > > > which then probably would run in each thread. > > > > It would cost you 1 more disk if you run 2 RAID5's > > > > so you get 8 TB payload out of your 12 GB total (6 drives of 2 TB each). > > > > > > > > Another way to get better performance could be to use less > > > > CPU-intensitive RAID types. RAID5 is intensitive as it needs to > > > > calculate XOR information all the time. Maybe a mirrored > > > > raid type like RAID10,f2 would give you less CPU usage, > > > > and the run 2 RAIDS to have it running in both hyperthreads. > > > > Here you would then only get 6 TB payload of your 12 GB disks, > > > > but then also probably a faster system. > > > > > > > > Best regards > > > > keld > > > > > > > > > > Hi, > > > > > > It's interesting what you say about the XOR calculations. I thought > > > that it was only calculated on writes? The Atom (330) has HT, so Linux > > > sees 4 logical CPUs. > > > > Yes you are right, it only calculates XOR on writes with RAID5. > > But then I am puzzled what all these CPU cycles are used for. > > Also many cycles are used on mirrored raid types. Why? > > Maybe some is because of LVM? I have been puzzled for a long time why > > ordinary RAID without LVM need to use so much CPU. Maybe a lot of data > > sguffling between buffers? Neil? > > What is your evidence that RAID1 uses lots of CPU? Much of this is raid10, but it should be the same: http://home.comcast.net/~jpiszcz/20080329-raid/ http://home.comcast.net/~jpiszcz/raid/20080528/raid-levels.html It seems like cpu usage is rather proportionate to the IO done. And the CPU usage does get up to about 40 % for reading, and 45 % for writing - this is most likely a significant delay factor. For slower CPUs like the Atom CPU this may be an even more significant delay factor. For RAID5 the situation is even worse, as expected. best regards keld -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance question, RAID5 2011-01-31 21:41 ` Keld Jørn Simonsen @ 2011-01-31 21:43 ` Roberto Spadim 0 siblings, 0 replies; 37+ messages in thread From: Roberto Spadim @ 2011-01-31 21:43 UTC (permalink / raw) To: Keld Jørn Simonsen Cc: NeilBrown, Mathias Burén, Stan Hoeppner, Roman Mamedov, CoolCold, Linux-RAID nice, but raid1 is not a very cpu consuming (a filesystem can use more cpu than a raid implementation...) a browser (firefox) too i think raid1 source code is well otimized for cpu and memory, maybe you need a faster cpu and not a less cpu consuming software, maybe a hardware raid could help you... 2011/1/31 Keld Jørn Simonsen <keld@keldix.com>: > On Tue, Feb 01, 2011 at 07:42:57AM +1100, NeilBrown wrote: >> On Mon, 31 Jan 2011 14:11:31 +0100 Keld Jørn Simonsen <keld@keldix.com> wrote: >> >> > On Mon, Jan 31, 2011 at 09:37:46AM +0000, Mathias Burén wrote: >> > > On 31 January 2011 08:52, Keld Jørn Simonsen <keld@keldix.com> wrote: >> > > > If your intallation is CPU bound, and you are >> > > > using an Atom N270 processor or the like, well some ideas: >> > > > >> > > > The Atom CPU may have threading, so you could run 2 RAIDs >> > > > which then probably would run in each thread. >> > > > It would cost you 1 more disk if you run 2 RAID5's >> > > > so you get 8 TB payload out of your 12 GB total (6 drives of 2 TB each). >> > > > >> > > > Another way to get better performance could be to use less >> > > > CPU-intensitive RAID types. RAID5 is intensitive as it needs to >> > > > calculate XOR information all the time. Maybe a mirrored >> > > > raid type like RAID10,f2 would give you less CPU usage, >> > > > and the run 2 RAIDS to have it running in both hyperthreads. >> > > > Here you would then only get 6 TB payload of your 12 GB disks, >> > > > but then also probably a faster system. >> > > > >> > > > Best regards >> > > > keld >> > > > >> > > >> > > Hi, >> > > >> > > It's interesting what you say about the XOR calculations. I thought >> > > that it was only calculated on writes? The Atom (330) has HT, so Linux >> > > sees 4 logical CPUs. >> > >> > Yes you are right, it only calculates XOR on writes with RAID5. >> > But then I am puzzled what all these CPU cycles are used for. >> > Also many cycles are used on mirrored raid types. Why? >> > Maybe some is because of LVM? I have been puzzled for a long time why >> > ordinary RAID without LVM need to use so much CPU. Maybe a lot of data >> > sguffling between buffers? Neil? >> >> What is your evidence that RAID1 uses lots of CPU? > > Much of this is raid10, but it should be the same: > http://home.comcast.net/~jpiszcz/20080329-raid/ > http://home.comcast.net/~jpiszcz/raid/20080528/raid-levels.html > > It seems like cpu usage is rather proportionate to the IO done. > And the CPU usage does get up to about 40 % for reading, and > 45 % for writing - this is most likely a significant delay factor. > For slower CPUs like the Atom CPU this may be an even more significant > delay factor. > > For RAID5 the situation is even worse, as expected. > > best regards > keld > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Roberto Spadim Spadim Technology / SPAEmpresarial -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 37+ messages in thread
end of thread, other threads:[~2011-02-01 14:32 UTC | newest] Thread overview: 37+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-01-29 22:48 Performance question, RAID5 Mathias Burén 2011-01-29 22:53 ` Roman Mamedov 2011-01-29 23:44 ` Mathias Burén 2011-01-29 23:57 ` Roman Mamedov 2011-01-30 0:15 ` Stan Hoeppner 2011-01-30 0:33 ` Mathias Burén 2011-01-30 0:27 ` Mathias Burén 2011-01-30 1:52 ` Keld Jørn Simonsen 2011-01-30 1:54 ` Mathias Burén 2011-01-30 5:56 ` Keld Jørn Simonsen 2011-01-30 12:12 ` Mathias Burén 2011-01-30 19:44 ` Stan Hoeppner 2011-01-30 19:46 ` Mathias Burén 2011-02-01 11:37 ` John Robinson 2011-02-01 13:53 ` Roberto Spadim 2011-02-01 14:02 ` Mathias Burén 2011-02-01 14:32 ` Roberto Spadim 2011-01-29 23:26 ` CoolCold 2011-01-30 0:18 ` Mathias Burén 2011-01-30 4:44 ` Roman Mamedov 2011-01-30 12:09 ` Mathias Burén 2011-01-30 12:15 ` Roman Mamedov 2011-01-30 19:41 ` Mathias Burén 2011-01-30 19:54 ` Roman Mamedov 2011-01-30 19:58 ` Mathias Burén 2011-01-30 20:03 ` Stan Hoeppner 2011-01-30 21:43 ` Mathias Burén 2011-01-31 3:39 ` Stan Hoeppner 2011-01-31 3:54 ` Roberto Spadim 2011-01-31 8:52 ` Keld Jørn Simonsen 2011-01-31 9:37 ` Mathias Burén 2011-01-31 13:11 ` Keld Jørn Simonsen 2011-01-31 14:43 ` Roberto Spadim 2011-01-31 18:44 ` Keld Jørn Simonsen 2011-01-31 20:42 ` NeilBrown 2011-01-31 21:41 ` Keld Jørn Simonsen 2011-01-31 21:43 ` Roberto Spadim
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.