Raid6 write performance

Raid6 write performance

[Peter Rabbitson]

Hi,

I am experimenting with raid6 on 4 drives on 2.6.27.11. The problem I am
having is that no matter what chunk size I use, the write benchmark
always comes out at single drive speed, although I should be seeing
double drive speed (read speed is at near 4x as expected). Is there some
hidden setting that I am overlooking, or is this a current known
limitation of raid6? In contrast if I make a raid5 on these 4 drives, I
get the expected 3xdrive write speed, and occasionally 4xdrive linear
read speed.

When the write test is running, I get about 14% of system cpu a sporadic
40% of iowait and the rest idle at all times (machine is in runlevel 1
so not to screw with results). Anyone has any ideas?

Thank you.

Re: Raid6 write performance

[thomas62186218]

I tested RAID 5 and RAID 6 with 12 x 15K SAS drives on Ubuntu 8.04 
64-bit and found their performance to be about the same. I used 256 K 
chunk size, v1.0 superblocks, stripecachesize of 16384, and readahead 
of 65536.

RAID 5 reads: 774 MB/sec
RAID 5 writes: 585 MB/sec

RAID 6 reads: 742 MB/sec
RAID 6 writes: 559 MB/sec

My CPU utilization remains under 10% though during writes, and I'm 
wondering what can be done to get write performance closer to read 
performance. I have dual quad-core CPUs so there's plenty of CPU to go 
around. Any ideas on that front?

-Thomas


-----Original Message-----
From: Peter Rabbitson <rabbit+list@rabbit.us>
To: linux-raid@vger.kernel.org
Sent: Sun, 18 Jan 2009 11:40 pm
Subject: Raid6 write performance










Hi,

I am experimenting with raid6 on 4 drives on 2.6.27.11. The problem I am
having is that no matter what chunk size I use, the write benchmark
always comes out at single drive speed, although I should be seeing
double drive speed (read speed is at near 4x as expected). Is there some
hidden setting that I am overlooking, or is this a current known
limitation of raid6? In contrast if I make a raid5 on these 4 drives, I
get the expected 3xdrive write speed, and occasionally 4xdrive linear
read speed.

When the write test is running, I get about 14% of system cpu a sporadic
40% of iowait and the rest idle at all times (machine is in runlevel 1
so not to screw with results). Anyone has any ideas?

Thank you.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid6 write performance

[NiftyFedora Mitch]

On Mon, Jan 19, 2009 at 12:10 AM,  <thomas62186218@aol.com> wrote:
>
> I tested RAID 5 and RAID 6 with 12 x 15K SAS drives on Ubuntu 8.04 64-bit
> and found their performance to be about the same. I used 256 K chunk size,
> v1.0 superblocks, stripecachesize of 16384, and readahead of 65536.
>
> RAID 5 reads: 774 MB/sec
> RAID 5 writes: 585 MB/sec
>
> RAID 6 reads: 742 MB/sec
> RAID 6 writes: 559 MB/sec
>
> My CPU utilization remains under 10% though during writes, and I'm wondering
> what can be done to get write performance closer to read performance. I have
> dual quad-core CPUs so there's plenty of CPU to go around. Any ideas on that
> front?
>
> -Thomas
>
>
> -----Original Message-----
> From: Peter Rabbitson <rabbit+list@rabbit.us>
> To: linux-raid@vger.kernel.org
> Sent: Sun, 18 Jan 2009 11:40 pm
> Subject: Raid6 write performance

> Hi,
>
> I am experimenting with raid6 on 4 drives on 2.6.27.11. The problem I am
> having is that no matter what chunk size I use, the write benchmark
> always comes out at single drive speed, although I should be seeing
> double drive speed (read speed is at near 4x as expected). Is there some
> hidden setting that I am overlooking, or is this a current known
> limitation of raid6? In contrast if I make a raid5 on these 4 drives, I
> get the expected 3xdrive write speed, and occasionally 4xdrive linear
> read speed.
>
> When the write test is running, I get about 14% of system cpu a sporadic
> 40% of iowait and the rest idle at all times (machine is in runlevel 1
> so not to screw with results). Anyone has any ideas?

Read bandwidth will always be quicker than writes with parity.
Data and parity both need to be written in an atomic way so the completion
will be gated by the last write to be "posted as done" back to the
system.

It can pay to locate the journal  on a physically different, smaller
and faster resource.
One informative experiment might be to mount the file system as ext2 and compare
and contrast with the same FS mounted as ext3.  I am not recommending ext2
over ext3 other than as an experiment to see what the impact of the
journal activity is...

-- 
        NiftyFedora
        T o m   M i t c h e l l

Re: Raid6 write performance

[Peter Rabbitson]

Peter Rabbitson wrote:
> Hi,
> 
> I am experimenting with raid6 on 4 drives on 2.6.27.11. The problem I am
> having is that no matter what chunk size I use, the write benchmark
> always comes out at single drive speed, although I should be seeing
> double drive speed (read speed is at near 4x as expected). 

Please disregard this, I appear to be having some sort of weird hardware
congestion. Can someone explain what could be going on?


Standalone read speed:
--------------------------------
root@Arzamas:~# for i in b c d e ; do dd if=/dev/sd${i}2 of=/dev/null
bs=32M count=32 ; done
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 10.4133 s, 103 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 10.4781 s, 102 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 10.4317 s, 103 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 10.1242 s, 106 MB/s
root@Arzamas:~#


Combined read speed - limited by the width of a 32bit 66mhz PCI (dev
02:04.0 below)
--------------------------------
root@Arzamas:~# for i in b c d e ; do dd if=/dev/sd${i}2 of=/dev/null
bs=32M count=32 & done
[1] 19491
[2] 19492
[3] 19493
[4] 19494
root@Arzamas:~# 32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 17.8422 s, 60.2 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 18.2496 s, 58.8 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 18.5542 s, 57.9 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 19.0774 s, 56.3 MB/s


Standalone write:
-------------------------------
root@Arzamas:~# for i in b c d e ; do dd if=/dev/zero of=/dev/sd${i}2
bs=32M count=32 ; done
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 10.7629 s, 99.8 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 10.753 s, 99.9 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 12.1836 s, 88.1 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 10.2433 s, 105 MB/s
root@Arzamas:~#


Combined write - half of the read speed???:
-------------------------------
root@Arzamas:~# for i in b c d e ; do dd if=/dev/zero of=/dev/sd${i}2
bs=32M count=32 & done
[1] 19506
[2] 19507
[3] 19508
[4] 19509
root@Arzamas:~# 32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 40.1631 s, 26.7 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 40.2603 s, 26.7 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 40.2862 s, 26.7 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 40.3178 s, 26.6 MB/s


Hardware:
---------------------------

root@Arzamas:~# lspci
00:00.0 Host bridge: Intel Corporation 82875P/E7210 Memory Controller
Hub (rev 02)
00:03.0 PCI bridge: Intel Corporation 82875P/E7210 Processor to PCI to
CSA Bridge (rev 02)
00:06.0 System peripheral: Intel Corporation 82875P/E7210 Processor to
I/O Memory Interface (rev 02)
00:1c.0 PCI bridge: Intel Corporation 6300ESB 64-bit PCI-X Bridge (rev 02)
00:1d.0 USB Controller: Intel Corporation 6300ESB USB Universal Host
Controller (rev 02)
00:1d.1 USB Controller: Intel Corporation 6300ESB USB Universal Host
Controller (rev 02)
00:1d.4 System peripheral: Intel Corporation 6300ESB Watchdog Timer (rev 02)
00:1d.5 PIC: Intel Corporation 6300ESB I/O Advanced Programmable
Interrupt Controller (rev 02)
00:1d.7 USB Controller: Intel Corporation 6300ESB USB2 Enhanced Host
Controller (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 0a)
00:1f.0 ISA bridge: Intel Corporation 6300ESB LPC Interface Controller
(rev 02)
00:1f.1 IDE interface: Intel Corporation 6300ESB PATA Storage Controller
(rev 02)
00:1f.3 SMBus: Intel Corporation 6300ESB SMBus Controller (rev 02)
01:01.0 Ethernet controller: Intel Corporation 82547GI Gigabit Ethernet
Controller
02:01.0 SCSI storage controller: Adaptec AIC-7901 U320 (rev 10)
02:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701
Gigabit Ethernet (rev 15)
02:03.0 Mass storage controller: Silicon Image, Inc. SiI 3124 PCI-X
Serial ATA Controller (rev 02)
02:04.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet
Controller (rev 05)
03:00.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
03:01.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet
Pro 100 (rev 10)
03:02.0 Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado]
(rev 78)
root@Arzamas:~#


root@Arzamas:~# lspci -v -s 02:
02:01.0 SCSI storage controller: Adaptec AIC-7901 U320 (rev 10)
	Subsystem: Intel Corporation Device 342f
	Flags: bus master, 66MHz, slow devsel, latency 64, IRQ 27
	I/O ports at b800 [disabled] [size=256]
	Memory at fc5fe000 (64-bit, non-prefetchable) [size=8K]
	I/O ports at b400 [disabled] [size=256]
	Expansion ROM at fc500000 [disabled] [size=512K]
	Capabilities: [dc] Power Management version 2
	Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1
Enable-
	Capabilities: [94] PCI-X non-bridge device
	Kernel driver in use: aic79xx

02:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701
Gigabit Ethernet (rev 15)
	Subsystem: Compaq Computer Corporation NC7770 Gigabit Server Adapter
(PCI-X, 10/100/1000-T)
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 24
	Memory at fc5e0000 (64-bit, non-prefetchable) [size=64K]
	[virtual] Expansion ROM at fc5d0000 [disabled] [size=64K]
	Capabilities: [40] PCI-X non-bridge device
	Capabilities: [48] Power Management version 2
	Capabilities: [50] Vital Product Data <?>
	Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
Enable-
	Kernel driver in use: tg3

02:03.0 Mass storage controller: Silicon Image, Inc. SiI 3124 PCI-X
Serial ATA Controller (rev 02)  <----- where the 4 drives are attached to
	Subsystem: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller
	Flags: bus master, stepping, 66MHz, medium devsel, latency 64, IRQ 25
	Memory at fc5fd800 (64-bit, non-prefetchable) [size=128]
	Memory at fc5f0000 (64-bit, non-prefetchable) [size=32K]
	I/O ports at bc00 [size=16]
	Expansion ROM at fc480000 [disabled] [size=512K]
	Capabilities: [64] Power Management version 2
	Capabilities: [40] PCI-X non-bridge device
	Capabilities: [54] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0
Enable-
	Kernel driver in use: sata_sil24

02:04.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet
Controller (rev 05)
	Subsystem: Intel Corporation PRO/1000 GT Desktop Adapter
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 26
	Memory at fc5a0000 (32-bit, non-prefetchable) [size=128K]
	Memory at fc580000 (32-bit, non-prefetchable) [size=128K]
	I/O ports at b000 [size=64]
	Expansion ROM at fc460000 [disabled] [size=128K]
	Capabilities: [dc] Power Management version 2
	Capabilities: [e4] PCI-X non-bridge device
	Kernel driver in use: e1000

root@Arzamas:~#

root@Arzamas:~# cat /proc/interrupts
           CPU0       CPU1
  0:        127          0   IO-APIC-edge      timer
  1:          8          0   IO-APIC-edge      i8042
  6:          3          0   IO-APIC-edge      floppy
  8:          2          0   IO-APIC-edge      rtc
 10:          0          0   IO-APIC-fasteoi   acpi
 12:        105          0   IO-APIC-edge      i8042
 14:        100          0   IO-APIC-edge      ata_piix
 15:          0          0   IO-APIC-edge      ata_piix
 16:     245660          0   IO-APIC-fasteoi   uhci_hcd:usb2, wan_a
 17:       5967          0   IO-APIC-fasteoi   lan_untr
 18:      92684          0   IO-APIC-fasteoi   lan_main
 19:          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
 23:          2          0   IO-APIC-fasteoi   ehci_hcd:usb1
 24:      73850          0   IO-APIC-fasteoi   wan_b
 25:    1032325          0   IO-APIC-fasteoi   sata_sil24
 26:       5494          0   IO-APIC-fasteoi   lan_srv
 27:         22          0   IO-APIC-fasteoi   aic79xx
NMI:          0          0   Non-maskable interrupts
LOC:    1383148    1381673   Local timer interrupts
RES:      52448      46803   Rescheduling interrupts
CAL:       2552       1883   function call interrupts
TLB:       8035      16978   TLB shootdowns
TRM:          0          0   Thermal event interrupts
SPU:          0          0   Spurious interrupts
ERR:          0
MIS:          0
root@Arzamas:~#

Re: Raid6 write performance

[Bernd Schubert]

Hello Peter,

On Monday 19 January 2009 11:37:11 Peter Rabbitson wrote:
> Peter Rabbitson wrote:
> > Hi,
> >
> > I am experimenting with raid6 on 4 drives on 2.6.27.11. The problem I am
> > having is that no matter what chunk size I use, the write benchmark
> > always comes out at single drive speed, although I should be seeing
> > double drive speed (read speed is at near 4x as expected).
>
> Please disregard this, I appear to be having some sort of weird hardware
> congestion. Can someone explain what could be going on?

I don't know what it is going on, but:


> Combined read speed - limited by the width of a 32bit 66mhz PCI (dev
> 02:04.0 below)

No! You you do have PCI-X, so 64bit with 66MHz. Maybe you are even luckily and 
it is as 133MHz, depends on how many other devices are connected to this bus. 
lspci will always say it is running at 66MHz, though, it also might run at 
100 or 133 MHz,

>
> 02:03.0 Mass storage controller: Silicon Image, Inc. SiI 3124 PCI-X
> Serial ATA Controller (rev 02)  <----- where the 4 drives are attached to
> 	Subsystem: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller
> 	Flags: bus master, stepping, 66MHz, medium devsel, latency 64, IRQ 25
> 	Memory at fc5fd800 (64-bit, non-prefetchable) [size=128]
> 	Memory at fc5f0000 (64-bit, non-prefetchable) [size=32K]
> 	I/O ports at bc00 [size=16]
> 	Expansion ROM at fc480000 [disabled] [size=512K]
> 	Capabilities: [64] Power Management version 2
> 	Capabilities: [40] PCI-X non-bridge device
> 	Capabilities: [54] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0


Cheers,
Bernd

-- 
Bernd Schubert
Q-Leap Networks GmbH

Re: Raid6 write performance

[Peter Rabbitson]

Bernd Schubert wrote:
> Hello Peter,
> 
> On Monday 19 January 2009 11:37:11 Peter Rabbitson wrote:
>> Peter Rabbitson wrote:
>>> Hi,
>>>
>>> I am experimenting with raid6 on 4 drives on 2.6.27.11. The problem I am
>>> having is that no matter what chunk size I use, the write benchmark
>>> always comes out at single drive speed, although I should be seeing
>>> double drive speed (read speed is at near 4x as expected).
>> Please disregard this, I appear to be having some sort of weird hardware
>> congestion. Can someone explain what could be going on?
> 
> I don't know what it is going on, but:
> 
> 
>> Combined read speed - limited by the width of a 32bit 66mhz PCI (dev
>> 02:04.0 below)
> 
> No! You you do have PCI-X, so 64bit with 66MHz. Maybe you are even luckily and 
> it is as 133MHz, depends on how many other devices are connected to this bus. 
> lspci will always say it is running at 66MHz, though, it also might run at 
> 100 or 133 MHz,
> 

And on the same bus (the last lspci) I have:

02:04.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet
Controller (rev 05)
	Subsystem: Intel Corporation PRO/1000 GT Desktop Adapter
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 26
	Memory at fc5a0000 (32-bit, non-prefetchable) [size=128K]
	Memory at fc580000 (32-bit, non-prefetchable) [size=128K]
	I/O ports at b000 [size=64]
	Expansion ROM at fc460000 [disabled] [size=128K]
	Capabilities: [dc] Power Management version 2
	Capabilities: [e4] PCI-X non-bridge device
	Kernel driver in use: e1000

which afaik brings everything to 32bits no?

Re: Raid6 write performance

[Bernd Schubert]

On Monday 19 January 2009 12:01:37 Peter Rabbitson wrote:
> Bernd Schubert wrote:
> > Hello Peter,
> >
> > On Monday 19 January 2009 11:37:11 Peter Rabbitson wrote:
> >> Peter Rabbitson wrote:
> >>> Hi,
> >>>
> >>> I am experimenting with raid6 on 4 drives on 2.6.27.11. The problem I
> >>> am having is that no matter what chunk size I use, the write benchmark
> >>> always comes out at single drive speed, although I should be seeing
> >>> double drive speed (read speed is at near 4x as expected).
> >>
> >> Please disregard this, I appear to be having some sort of weird hardware
> >> congestion. Can someone explain what could be going on?
> >
> > I don't know what it is going on, but:
> >> Combined read speed - limited by the width of a 32bit 66mhz PCI (dev
> >> 02:04.0 below)
> >
> > No! You you do have PCI-X, so 64bit with 66MHz. Maybe you are even
> > luckily and it is as 133MHz, depends on how many other devices are
> > connected to this bus. lspci will always say it is running at 66MHz,
> > though, it also might run at 100 or 133 MHz,
>
> And on the same bus (the last lspci) I have:
>
> 02:04.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet
> Controller (rev 05)
> 	Subsystem: Intel Corporation PRO/1000 GT Desktop Adapter
> 	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 26
> 	Memory at fc5a0000 (32-bit, non-prefetchable) [size=128K]
> 	Memory at fc580000 (32-bit, non-prefetchable) [size=128K]
> 	I/O ports at b000 [size=64]
> 	Expansion ROM at fc460000 [disabled] [size=128K]
> 	Capabilities: [dc] Power Management version 2
> 	Capabilities: [e4] PCI-X non-bridge device
> 	Kernel driver in use: e1000
>
> which afaik brings everything to 32bits no?

Hmm, I don't know what happens if a 32-bit card is in a 64bit PCI-X bus. 
Wouldn't it only use half of the available lines for this card and all lines 
for the 64bit cards?  Hmm, on the other hand your maximum combined rate seems 
to be at around 240 MB/s, which is close to 32bit x 66 MHz.Any chance you can 
remove this NIC and retest?


Cheers,
Bernd

-- 
Bernd Schubert
Q-Leap Networks GmbH

Re: Raid6 write performance

[Keld Jørn Simonsen]

On Mon, Jan 19, 2009 at 03:10:01AM -0500, thomas62186218@aol.com wrote:
> 
> I tested RAID 5 and RAID 6 with 12 x 15K SAS drives on Ubuntu 8.04 
> 64-bit and found their performance to be about the same. I used 256 K 
> chunk size, v1.0 superblocks, stripecachesize of 16384, and readahead 
> of 65536.
> 
> RAID 5 reads: 774 MB/sec
> RAID 5 writes: 585 MB/sec
> 
> RAID 6 reads: 742 MB/sec
> RAID 6 writes: 559 MB/sec
> 
> My CPU utilization remains under 10% though during writes, and I'm 
> wondering what can be done to get write performance closer to read 
> performance. I have dual quad-core CPUs so there's plenty of CPU to go 
> around. Any ideas on that front?

Seems like it is the same equipment that you also did the raid10,f2
tests for. I think it would be interesting to have a consolidated list
of performance comparisons, when you have completed your tweakings.
Is that part of your plan?

best regards
keld

Re: Raid6 write performance

[Peter Rabbitson]

Keld Jørn Simonsen wrote:
> On Mon, Jan 19, 2009 at 03:10:01AM -0500, thomas62186218@aol.com wrote:
>> I tested RAID 5 and RAID 6 with 12 x 15K SAS drives on Ubuntu 8.04 
>> 64-bit and found their performance to be about the same. I used 256 K 
>> chunk size, v1.0 superblocks, stripecachesize of 16384, and readahead 
>> of 65536.
>>
>> RAID 5 reads: 774 MB/sec
>> RAID 5 writes: 585 MB/sec
>>
>> RAID 6 reads: 742 MB/sec
>> RAID 6 writes: 559 MB/sec
>>
>> My CPU utilization remains under 10% though during writes, and I'm 
>> wondering what can be done to get write performance closer to read 
>> performance. I have dual quad-core CPUs so there's plenty of CPU to go 
>> around. Any ideas on that front?
> 
> Seems like it is the same equipment that you also did the raid10,f2
> tests for. I think it would be interesting to have a consolidated list
> of performance comparisons, when you have completed your tweakings.
> Is that part of your plan?
> 
> best regards
> keld

Not on this machine, but yes, I am planning to do something like this on
my workstation. I need to get my drives/arrays in order first (got
bitten by the 7200.11 fiasco).
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid6 write performance

[Bill Davidsen]

NiftyFedora Mitch wrote:
> On Mon, Jan 19, 2009 at 12:10 AM,  <thomas62186218@aol.com> wrote:
>   
>> I tested RAID 5 and RAID 6 with 12 x 15K SAS drives on Ubuntu 8.04 64-bit
>> and found their performance to be about the same. I used 256 K chunk size,
>> v1.0 superblocks, stripecachesize of 16384, and readahead of 65536.
>>
>> RAID 5 reads: 774 MB/sec
>> RAID 5 writes: 585 MB/sec
>>
>> RAID 6 reads: 742 MB/sec
>> RAID 6 writes: 559 MB/sec
>>
>> My CPU utilization remains under 10% though during writes, and I'm wondering
>> what can be done to get write performance closer to read performance. I have
>> dual quad-core CPUs so there's plenty of CPU to go around. Any ideas on that
>> front?
>>
>> -Thomas
>>
>>
>> -----Original Message-----
>> From: Peter Rabbitson <rabbit+list@rabbit.us>
>> To: linux-raid@vger.kernel.org
>> Sent: Sun, 18 Jan 2009 11:40 pm
>> Subject: Raid6 write performance
>>     
>
>   
>> Hi,
>>
>> I am experimenting with raid6 on 4 drives on 2.6.27.11. The problem I am
>> having is that no matter what chunk size I use, the write benchmark
>> always comes out at single drive speed, although I should be seeing
>> double drive speed (read speed is at near 4x as expected). Is there some
>> hidden setting that I am overlooking, or is this a current known
>> limitation of raid6? In contrast if I make a raid5 on these 4 drives, I
>> get the expected 3xdrive write speed, and occasionally 4xdrive linear
>> read speed.
>>
>> When the write test is running, I get about 14% of system cpu a sporadic
>> 40% of iowait and the rest idle at all times (machine is in runlevel 1
>> so not to screw with results). Anyone has any ideas?
>>     
>
> Read bandwidth will always be quicker than writes with parity.
> Data and parity both need to be written in an atomic way so the completion
> will be gated by the last write to be "posted as done" back to the
> system.
>
> It can pay to locate the journal  on a physically different, smaller
> and faster resource.
> One informative experiment might be to mount the file system as ext2 and compare
> and contrast with the same FS mounted as ext3.  I am not recommending ext2
> over ext3 other than as an experiment to see what the impact of the
> journal activity is...
>
>   
Also see noatime, relatime, etc.

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

Re: Raid6 write performance

[H. Peter Anvin]

Peter Rabbitson wrote:
> Hi,
> 
> I am experimenting with raid6 on 4 drives on 2.6.27.11. The problem I am
> having is that no matter what chunk size I use, the write benchmark
> always comes out at single drive speed, although I should be seeing
> double drive speed (read speed is at near 4x as expected).

I have no idea why you "should" be seeing double drive speed.  All
drives have to be written, so you'd logically see single drive speed.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: Raid6 write performance

[Michał Przyłuski]

Hello,

2009/2/28 H. Peter Anvin <hpa@zytor.com>:
> Peter Rabbitson wrote:
>> Hi,
>>
>> I am experimenting with raid6 on 4 drives on 2.6.27.11. The problem I am
>> having is that no matter what chunk size I use, the write benchmark
>> always comes out at single drive speed, although I should be seeing
>> double drive speed (read speed is at near 4x as expected).
>
> I have no idea why you "should" be seeing double drive speed.  All
> drives have to be written, so you'd logically see single drive speed.

I'm afraid that might be incorrect.

Let's assume we want to write 100MB of data onto a 4 drive raid6.
Let's divide 100MB of data into two parts, say A and B, each 50MB big.
Writing the data on the raid, would mean writing:
* A on disk1
* B on disk2
* XOR(A,B) on disk3
* Q(A,B) on disk4
That is actually assuming 50MB chunk, and whole chunk writes, etc.
Each of written portions would have been 50MB in size. That sounds
reasonable to me, as with 2 data disks, only half of data has to be
written on each. The fact that disks are really striped with data, XOR
and Q doesn't change the image in terms of amount written.

I do hope I had understood the situation correctly, but I'll be ever
happy to be proved wrong.

Kind regards,
Mike

Re: Raid6 write performance

[Peter Rabbitson]

H. Peter Anvin wrote:
> Peter Rabbitson wrote:
>> Hi,
>>
>> I am experimenting with raid6 on 4 drives on 2.6.27.11. The problem I am
>> having is that no matter what chunk size I use, the write benchmark
>> always comes out at single drive speed, although I should be seeing
>> double drive speed (read speed is at near 4x as expected).
> 
> I have no idea why you "should" be seeing double drive speed.  All
> drives have to be written, so you'd logically see single drive speed.
> 

Because with properly adjusted elevators and chunk sizes it is reasonable
to expect N * S write speed from _any_ raid, where N is the number of
different data bearing disks in a stripe, and S is the speed of a hard
drive (assuming the drive speeds are equal). So for raid5 we have N =
numdisks-1, for raid6 numdisks-2, for raid10 -n4 -pf3 we get 4-(3-1) and
so on. I have personally verified the write behavior for raid10 and raid5,
don't see why it should/would be different for raid6.

In any case the witnessed problem was due to a hardware misconfiguration,
which has not been resolved to this day. Thus thread is dead :)

Re: Raid6 write performance

[H. Peter Anvin]

Michał Przyłuski wrote:
> 
> I'm afraid that might be incorrect.
> 
> Let's assume we want to write 100MB of data onto a 4 drive raid6.
> Let's divide 100MB of data into two parts, say A and B, each 50MB big.
> Writing the data on the raid, would mean writing:
> * A on disk1
> * B on disk2
> * XOR(A,B) on disk3
> * Q(A,B) on disk4
> That is actually assuming 50MB chunk, and whole chunk writes, etc.
> Each of written portions would have been 50MB in size. That sounds
> reasonable to me, as with 2 data disks, only half of data has to be
> written on each. The fact that disks are really striped with data, XOR
> and Q doesn't change the image in terms of amount written.
> 
> I do hope I had understood the situation correctly, but I'll be ever
> happy to be proved wrong.
> 

Ah, sorry, yes you're of course right.  I was thinking about latency,
not throughput, for some idiotic reason.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid6 write performance

[Bill Davidsen]

Peter Rabbitson wrote:
> H. Peter Anvin wrote:
>   
>> Peter Rabbitson wrote:
>>     
>>> Hi,
>>>
>>> I am experimenting with raid6 on 4 drives on 2.6.27.11. The problem I am
>>> having is that no matter what chunk size I use, the write benchmark
>>> always comes out at single drive speed, although I should be seeing
>>> double drive speed (read speed is at near 4x as expected).
>>>       
>> I have no idea why you "should" be seeing double drive speed.  All
>> drives have to be written, so you'd logically see single drive speed.
>>
>>     
>
> Because with properly adjusted elevators and chunk sizes it is reasonable
> to expect N * S write speed from _any_ raid, where N is the number of
> different data bearing disks in a stripe, and S is the speed of a hard
> drive (assuming the drive speeds are equal). So for raid5 we have N =
> numdisks-1, for raid6 numdisks-2, for raid10 -n4 -pf3 we get 4-(3-1) and
> so on. I have personally verified the write behavior for raid10 and raid5,
> don't see why it should/would be different for raid6.
>   

That's a lovely theory, but in practice I have to say I have never 
measured any such thing, using benchmarks intended to match real world, 
or even heavy disk writes of a dumb nature like dd. I have tested 
through the raw device, and through filesystems, tuned stripe-cache-size 
and buffers, tried setting "stride" in ext3, all to conclude that with 
raid5 I see essentially write speed of 1x a single drive and read speed 
of (N-1)x as you suggest. Actually, looking at results for arrays with 
more drives I can see a trend to write at (N/3)x speed, being a 
seek-write for full chunk data and seef-read-write for XOR. But even on 
six drive arrays I don't get near (N-1)x in measurable.

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark