performance of raid5 on fast devices

performance of raid5 on fast devices

[Jake Yao]

I have a raid5 array on 4 NVMe drives, and the performance on the
array is only marginally better than a single drive. Unlike a similar
raid5 array on 4 SAS SSD or HDD,  the performance on array is 3x
better than a single drive, which is expected.

It looks like when the single kernel thread associated with the raid
device running at 100%, the array performance hit its peak. This can
happen easily for fast devices like NVMe.

This can reproduced by creating a raid5 with 4 ramdisks as well, and
comparing performance on the array and one ramdisk. Sometimes the
performance on the array is worse than a single ramdisk.

The kernel version is 4.9.0-rc3 and mdadm is release 3.4, no write
journal is configured.

Is this a known issue?

Please cc me on the email as I am not on the mail list.

Thanks!

Re: performance of raid5 on fast devices

[Stan Hoeppner]

On 01/16/2017 08:35 PM, Jake Yao wrote:
> I have a raid5 array on 4 NVMe drives, and the performance on the
> array is only marginally better than a single drive. Unlike a similar
> raid5 array on 4 SAS SSD or HDD,  the performance on array is 3x
> better than a single drive, which is expected.
>
> It looks like when the single kernel thread associated with the raid
> device running at 100%, the array performance hit its peak. This can
> happen easily for fast devices like NVMe.
The md raid personalities are limited to a single kernel write thread.  
Work is in progress to alleviate this bottleneck by using multiple write 
threads.  When it will hit mainline I don't know.

> This can reproduced by creating a raid5 with 4 ramdisks as well, and
> comparing performance on the array and one ramdisk. Sometimes the
> performance on the array is worse than a single ramdisk.
>
> The kernel version is 4.9.0-rc3 and mdadm is release 3.4, no write
> journal is configured.
>
> Is this a known issue?
>
> Please cc me on the email as I am not on the mail list.
>
> Thanks!

Re: performance of raid5 on fast devices

[Coly Li]

On 2017/1/17 上午11:10, Stan Hoeppner wrote:
> On 01/16/2017 08:35 PM, Jake Yao wrote:
>> I have a raid5 array on 4 NVMe drives, and the performance on the
>> array is only marginally better than a single drive. Unlike a similar
>> raid5 array on 4 SAS SSD or HDD,  the performance on array is 3x
>> better than a single drive, which is expected.
>>
>> It looks like when the single kernel thread associated with the raid
>> device running at 100%, the array performance hit its peak. This can
>> happen easily for fast devices like NVMe.
> The md raid personalities are limited to a single kernel write thread. 
> Work is in progress to alleviate this bottleneck by using multiple write
> threads.  When it will hit mainline I don't know.

If you want 8 writing threads, and your md raid5 device is /dev/md0, you
may have a try with,
	echo 8 > /sys/block/md0/md/group_thread_cnt

> 
>> This can reproduced by creating a raid5 with 4 ramdisks as well, and
>> comparing performance on the array and one ramdisk. Sometimes the
>> performance on the array is worse than a single ramdisk.
>>
>> The kernel version is 4.9.0-rc3 and mdadm is release 3.4, no write
>> journal is configured.
>>
>> Is this a known issue?

It was, but you are on 4.9 kernel, group_thread_cnt should work for you.

Coly

Re: performance of raid5 on fast devices

[Roman Mamedov]

On Mon, 16 Jan 2017 21:35:21 -0500
Jake Yao <jgyao1@gmail.com> wrote:

> I have a raid5 array on 4 NVMe drives, and the performance on the
> array is only marginally better than a single drive. Unlike a similar
> raid5 array on 4 SAS SSD or HDD,  the performance on array is 3x
> better than a single drive, which is expected.
> 
> It looks like when the single kernel thread associated with the raid
> device running at 100%, the array performance hit its peak. This can
> happen easily for fast devices like NVMe.
> 
> This can reproduced by creating a raid5 with 4 ramdisks as well, and
> comparing performance on the array and one ramdisk. Sometimes the
> performance on the array is worse than a single ramdisk.
> 
> The kernel version is 4.9.0-rc3 and mdadm is release 3.4, no write
> journal is configured.
> 
> Is this a known issue?

How do you measure the performance?

Sure it may be CPU-bound in the end, but also why not try the usual
optimization tricks, such as:

  * increase your stripe_cache_size, it's not uncommon that this can speed up
    linear writes by as much as several times;

  * if you meant reads, you could look into read-ahead settings for the array;

  * and in both cases, try experimenting with different stripe sizes (if you
    were using 512K, try with 64K stripes).

-- 
With respect,
Roman

Re: performance of raid5 on fast devices

[Jake Yao]

Thanks for the response.

It helps a little by increasing group_thread_cnt, but not to the
extend of 3x expected.  It looks like the single kernel thread is
still the bottleneck.

On Tue, Jan 17, 2017 at 12:04 AM, Coly Li <colyli@suse.de> wrote:
> On 2017/1/17 上午11:10, Stan Hoeppner wrote:
>> On 01/16/2017 08:35 PM, Jake Yao wrote:
>>> I have a raid5 array on 4 NVMe drives, and the performance on the
>>> array is only marginally better than a single drive. Unlike a similar
>>> raid5 array on 4 SAS SSD or HDD,  the performance on array is 3x
>>> better than a single drive, which is expected.
>>>
>>> It looks like when the single kernel thread associated with the raid
>>> device running at 100%, the array performance hit its peak. This can
>>> happen easily for fast devices like NVMe.
>> The md raid personalities are limited to a single kernel write thread.
>> Work is in progress to alleviate this bottleneck by using multiple write
>> threads.  When it will hit mainline I don't know.
>
> If you want 8 writing threads, and your md raid5 device is /dev/md0, you
> may have a try with,
>         echo 8 > /sys/block/md0/md/group_thread_cnt
>
>>
>>> This can reproduced by creating a raid5 with 4 ramdisks as well, and
>>> comparing performance on the array and one ramdisk. Sometimes the
>>> performance on the array is worse than a single ramdisk.
>>>
>>> The kernel version is 4.9.0-rc3 and mdadm is release 3.4, no write
>>> journal is configured.
>>>
>>> Is this a known issue?
>
> It was, but you are on 4.9 kernel, group_thread_cnt should work for you.
>
> Coly
>

Re: performance of raid5 on fast devices

[Jake Yao]

Thanks for the response.

I am using fio for performance measurement.

The chunk size of raid5 array is 32K, and the block size in fio is set
to 96K(3x chunk size) which is also the optimal_io_size, ioengine is
set to libaio with direct IO.

Increasing stripe_cache_size does not help much, and it looks like the
write is limited by the single kernel thread as mentioned earlier.


On Tue, Jan 17, 2017 at 12:10 AM, Roman Mamedov <rm@romanrm.net> wrote:
> On Mon, 16 Jan 2017 21:35:21 -0500
> Jake Yao <jgyao1@gmail.com> wrote:
>
>> I have a raid5 array on 4 NVMe drives, and the performance on the
>> array is only marginally better than a single drive. Unlike a similar
>> raid5 array on 4 SAS SSD or HDD,  the performance on array is 3x
>> better than a single drive, which is expected.
>>
>> It looks like when the single kernel thread associated with the raid
>> device running at 100%, the array performance hit its peak. This can
>> happen easily for fast devices like NVMe.
>>
>> This can reproduced by creating a raid5 with 4 ramdisks as well, and
>> comparing performance on the array and one ramdisk. Sometimes the
>> performance on the array is worse than a single ramdisk.
>>
>> The kernel version is 4.9.0-rc3 and mdadm is release 3.4, no write
>> journal is configured.
>>
>> Is this a known issue?
>
> How do you measure the performance?
>
> Sure it may be CPU-bound in the end, but also why not try the usual
> optimization tricks, such as:
>
>   * increase your stripe_cache_size, it's not uncommon that this can speed up
>     linear writes by as much as several times;
>
>   * if you meant reads, you could look into read-ahead settings for the array;
>
>   * and in both cases, try experimenting with different stripe sizes (if you
>     were using 512K, try with 64K stripes).
>
> --
> With respect,
> Roman

Re: performance of raid5 on fast devices

[Heinz Mauelshagen]

Jake et al,

I took the oportunity to measure raid5 on a 4x NVME here with
variations of group_thread_cnt={0..10} minimal
stripe_cache_size={256,512,1024,2048,4096,8192,16384,32768}

This is on an X-99 with Intel E5-2640 and kernel 4.9.3-200.fc25.x86_64.

Highest active stripe count logged < 17K.


fio job/sections used:
----------------------------
[r-md0]
ioengine=libaio
iodepth=40
rw=read
bs=4096K
direct=1
size=4G
numjobs=8
filename=/dev/md0

[w-md0]
ioengine=libaio
iodepth=40
rw=write
bs=4096K
direct=1
size=4G
numjobs=8
filename=/dev/md0


Baseline performance seen with raid0:
---------------------------------------------------
md0 : active raid0 dm-350[3] dm-349[2] dm-348[1] dm-347[0]
       33521664 blocks super 1.2 32k chunks

READ: io=32768MB, aggrb=8202.3MB/s, minb=1025.3MB/s, maxb=1217.7MB/s, 
mint=3364msec, maxt=3995msec
WRITE: io=32768MB, aggrb=5746.8MB/s, minb=735584KB/s, maxb=836685KB/s, 
mint=5013msec, maxt=5702msec


Performance with raid5:
--------------------------------
md0 : active raid5 dm-350[3] dm-349[2] dm-348[1] dm-347[0]
       25141248 blocks super 1.2 level 5, 32k chunk, algorithm 2 [4/4] 
[UUUU]


READ: io=32768MB, aggrb=7375.3MB/s, minb=944025KB/s, maxb=1001.1MB/s, 
mint=4088msec, maxt=4443msec


Write results for group_thread_cnt/stripe_cache_size variations:
------------------------------------------------------------------------------------
0/256  -> WRITE: io=32768MB, aggrb=1296.4MB/s, minb=165927KB/s, 
maxb=167644KB/s, mint=25019msec, maxt=25278msec
1/256  -> WRITE: io=32768MB, aggrb=2152.6MB/s, minb=275524KB/s, 
maxb=278654KB/s, mint=15052msec, maxt=15223msec
2/256  -> WRITE: io=32768MB, aggrb=3177.4MB/s, minb=406700KB/s, 
maxb=415854KB/s, mint=10086msec, maxt=10313msec
3/256  -> WRITE: io=32768MB, aggrb=4026.6MB/s, minb=515397KB/s, 
maxb=524222KB/s, mint=8001msec, maxt=8138msec
4/256  -> WRITE: io=32768MB, aggrb=4172.2MB/s, minb=534034KB/s, 
maxb=552609KB/s, mint=7590msec, maxt=7854msec  *
5/256  -> WRITE: io=32768MB, aggrb=4166.9MB/s, minb=533355KB/s, 
maxb=547845KB/s, mint=7656msec, maxt=7864msec
6/256  -> WRITE: io=32768MB, aggrb=4189.3MB/s, minb=536218KB/s, 
maxb=556126KB/s, mint=7542msec, maxt=7822msec
7/256  -> WRITE: io=32768MB, aggrb=4192.5MB/s, minb=536630KB/s, 
maxb=560810KB/s, mint=7479msec, maxt=7816msec
8/256  -> WRITE: io=32768MB, aggrb=4185.2MB/s, minb=535807KB/s, 
maxb=562389KB/s, mint=7458msec, maxt=7828msec
9/256  -> WRITE: io=32768MB, aggrb=4192.1MB/s, minb=536699KB/s, 
maxb=577966KB/s, mint=7257msec, maxt=7815msec
10/256 -> WRITE: io=32768MB, aggrb=4182.3MB/s, minb=535329KB/s, 
maxb=568256KB/s, mint=7381msec, maxt=7835msec

0/512 -> WRITE: io=32768MB, aggrb=1297.8MB/s, minb=166025KB/s, 
maxb=167664KB/s, mint=25016msec, maxt=25263msec
1/512 -> WRITE: io=32768MB, aggrb=2148.5MB/s, minb=275000KB/s, 
maxb=278044KB/s, mint=15085msec, maxt=15252msec
2/512 -> WRITE: io=32768MB, aggrb=3158.4MB/s, minb=404270KB/s, 
maxb=411407KB/s, mint=10195msec, maxt=10375msec
3/512 -> WRITE: io=32768MB, aggrb=4102.7MB/s, minb=525141KB/s, 
maxb=539738KB/s, mint=7771msec, maxt=7987msec
4/512 -> WRITE: io=32768MB, aggrb=4162.8MB/s, minb=532745KB/s, 
maxb=541759KB/s, mint=7742msec, maxt=7873msec     *
5/512 -> WRITE: io=32768MB, aggrb=4178.6MB/s, minb=534851KB/s, 
maxb=549856KB/s, mint=7628msec, maxt=7842msec
6/512 -> WRITE: io=32768MB, aggrb=4167.4MB/s, minb=533422KB/s, 
maxb=562314KB/s, mint=7459msec, maxt=7863msec
7/512 -> WRITE: io=32768MB, aggrb=4192.1MB/s, minb=536699KB/s, 
maxb=566338KB/s, mint=7406msec, maxt=7815msec
8/512 -> WRITE: io=32768MB, aggrb=4189.8MB/s, minb=536287KB/s, 
maxb=558644KB/s, mint=7508msec, maxt=7821msec
9/512 -> WRITE: io=32768MB, aggrb=4165.8MB/s, minb=533219KB/s, 
maxb=559837KB/s, mint=7492msec, maxt=7866msec
10/512 -> WRITE: io=32768MB, aggrb=4177.2MB/s, minb=534783KB/s, 
maxb=570188KB/s, mint=7356msec, maxt=7843msec

0/1024 -> WRITE: io=32768MB, aggrb=1288.6MB/s, minb=164935KB/s, 
maxb=166877KB/s, mint=25134msec, maxt=25430msec
1/1024 -> WRITE: io=32768MB, aggrb=2218.5MB/s, minb=283955KB/s, 
maxb=289842KB/s, mint=14471msec, maxt=14771msec
2/1024 -> WRITE: io=32768MB, aggrb=3186.1MB/s, minb=407926KB/s, 
maxb=420903KB/s, mint=9965msec, maxt=10282msec
3/1024 -> WRITE: io=32768MB, aggrb=4107.4MB/s, minb=525733KB/s, 
maxb=538836KB/s, mint=7784msec, maxt=7978msec
4/1024 -> WRITE: io=32768MB, aggrb=4146.9MB/s, minb=530790KB/s, 
maxb=550505KB/s, mint=7619msec, maxt=7902msec
5/1024 -> WRITE: io=32768MB, aggrb=4160.5MB/s, minb=532542KB/s, 
maxb=550795KB/s, mint=7615msec, maxt=7876msec  *
6/1024 -> WRITE: io=32768MB, aggrb=4174.3MB/s, minb=534306KB/s, 
maxb=558942KB/s, mint=7504msec, maxt=7850msec
7/1024 -> WRITE: io=32768MB, aggrb=4189.8MB/s, minb=536287KB/s, 
maxb=556864KB/s, mint=7532msec, maxt=7821msec
8/1024 -> WRITE: io=32768MB, aggrb=4188.2MB/s, minb=536081KB/s, 
maxb=561035KB/s, mint=7476msec, maxt=7824msec
9/1024 -> WRITE: io=32768MB, aggrb=4167.4MB/s, minb=533422KB/s, 
maxb=567872KB/s, mint=7386msec, maxt=7863msec
10/1024 -> WRITE: io=32768MB, aggrb=4188.2MB/s, minb=536081KB/s, 
maxb=569878KB/s, mint=7360msec, maxt=7824msec

0/2048 -> WRITE: io=32768MB, aggrb=1265.7MB/s, minb=162004KB/s, 
maxb=166111KB/s, mint=25250msec, maxt=25890msec
1/2048 -> WRITE: io=32768MB, aggrb=2239.5MB/s, minb=286652KB/s, 
maxb=290846KB/s, mint=14421msec, maxt=14632msec
2/2048 -> WRITE: io=32768MB, aggrb=3184.5MB/s, minb=407609KB/s, 
maxb=413150KB/s, mint=10152msec, maxt=10290msec
3/2048 -> WRITE: io=32768MB, aggrb=4213.5MB/s, minb=539321KB/s, 
maxb=557901KB/s, mint=7518msec, maxt=7777msec     *
4/2048 -> WRITE: io=32768MB, aggrb=4168.5MB/s, minb=533558KB/s, 
maxb=543162KB/s, mint=7722msec, maxt=7861msec
5/2048 -> WRITE: io=32768MB, aggrb=4185.5MB/s, minb=535739KB/s, 
maxb=549352KB/s, mint=7635msec, maxt=7829msec
6/2048 -> WRITE: io=32768MB, aggrb=4181.8MB/s, minb=535260KB/s, 
maxb=553338KB/s, mint=7580msec, maxt=7836msec
7/2048 -> WRITE: io=32768MB, aggrb=4215.7MB/s, minb=539599KB/s, 
maxb=566109KB/s, mint=7409msec, maxt=7773msec
8/2048 -> WRITE: io=32768MB, aggrb=4200.5MB/s, minb=537662KB/s, 
maxb=568102KB/s, mint=7383msec, maxt=7801msec
9/2048 -> WRITE: io=32768MB, aggrb=4184.1MB/s, minb=535671KB/s, 
maxb=574483KB/s, mint=7301msec, maxt=7830msec
10/2048 -> WRITE: io=32768MB, aggrb=4172.7MB/s, minb=534102KB/s, 
maxb=567641KB/s, mint=7389msec, maxt=7853msec

0/4096 -> WRITE: io=32768MB, aggrb=1264.8MB/s, minb=161879KB/s, 
maxb=168588KB/s, mint=24879msec, maxt=25910msec
1/4096 -> WRITE: io=32768MB, aggrb=2349.4MB/s, minb=300710KB/s, 
maxb=312541KB/s, mint=13420msec, maxt=13948msec
2/4096 -> WRITE: io=32768MB, aggrb=3387.6MB/s, minb=433609KB/s, 
maxb=441877KB/s, mint=9492msec, maxt=9673msec
3/4096 -> WRITE: io=32768MB, aggrb=4182.3MB/s, minb=535329KB/s, 
maxb=552390KB/s, mint=7593msec, maxt=7835msec    *
4/4096 -> WRITE: io=32768MB, aggrb=4170.2MB/s, minb=533762KB/s, 
maxb=560061KB/s, mint=7489msec, maxt=7858msec
5/4096 -> WRITE: io=32768MB, aggrb=4179.6MB/s, minb=534919KB/s, 
maxb=548490KB/s, mint=7647msec, maxt=7841msec
6/4096 -> WRITE: io=32768MB, aggrb=4183.4MB/s, minb=535465KB/s, 
maxb=549208KB/s, mint=7637msec, maxt=7833msec
7/4096 -> WRITE: io=32768MB, aggrb=4174.9MB/s, minb=534374KB/s, 
maxb=557530KB/s, mint=7523msec, maxt=7849msec
8/4096 -> WRITE: io=32768MB, aggrb=4178.6MB/s, minb=534851KB/s, 
maxb=570188KB/s, mint=7356msec, maxt=7842msec
9/4096 -> WRITE: io=32768MB, aggrb=4180.2MB/s, minb=535056KB/s, 
maxb=570110KB/s, mint=7357msec, maxt=7839msec
10/4096 -> WRITE: io=32768MB, aggrb=4183.9MB/s, minb=535534KB/s, 
maxb=574640KB/s, mint=7299msec, maxt=7832msec

0/8192 -> WRITE: io=32768MB, aggrb=1260.9MB/s, minb=161381KB/s, 
maxb=171511KB/s, mint=24455msec, maxt=25990msec
1/8192 -> WRITE: io=32768MB, aggrb=2368.5MB/s, minb=303166KB/s, 
maxb=320444KB/s, mint=13089msec, maxt=13835msec
2/8192 -> WRITE: io=32768MB, aggrb=3408.8MB/s, minb=436225KB/s, 
maxb=458544KB/s, mint=9147msec, maxt=9615msec
3/8192 -> WRITE: io=32768MB, aggrb=4219.5MB/s, minb=540085KB/s, 
maxb=564585KB/s, mint=7429msec, maxt=7766msec     *
4/8192 -> WRITE: io=32768MB, aggrb=4208.6MB/s, minb=538698KB/s, 
maxb=570653KB/s, mint=7350msec, maxt=7786msec
5/8192 -> WRITE: io=32768MB, aggrb=4200.5MB/s, minb=537662KB/s, 
maxb=562013KB/s, mint=7463msec, maxt=7801msec
6/8192 -> WRITE: io=32768MB, aggrb=4189.3MB/s, minb=536218KB/s, 
maxb=585387KB/s, mint=7165msec, maxt=7822msec
7/8192 -> WRITE: io=32768MB, aggrb=4184.5MB/s, minb=535602KB/s, 
maxb=579323KB/s, mint=7240msec, maxt=7831msec
8/8192 -> WRITE: io=32768MB, aggrb=4186.6MB/s, minb=535876KB/s, 
maxb=572132KB/s, mint=7331msec, maxt=7827msec
9/8192 -> WRITE: io=32768MB, aggrb=4176.5MB/s, minb=534578KB/s, 
maxb=598246KB/s, mint=7011msec, maxt=7846msec
10/8192 -> WRITE: io=32768MB, aggrb=4184.1MB/s, minb=535671KB/s, 
maxb=580285KB/s, mint=7228msec, maxt=7830msec

0/16384 -> WRITE: io=32768MB, aggrb=1281.0MB/s, minb=163968KB/s, 
maxb=183542KB/s, mint=22852msec, maxt=25580msec
1/16384 -> WRITE: io=32768MB, aggrb=2451.8MB/s, minb=313827KB/s, 
maxb=337787KB/s, mint=12417msec, maxt=13365msec
2/16384 -> WRITE: io=32768MB, aggrb=3409.5MB/s, minb=436406KB/s, 
maxb=468532KB/s, mint=8952msec, maxt=9611msec
3/16384 -> WRITE: io=32768MB, aggrb=4192.5MB/s, minb=536630KB/s, 
maxb=566721KB/s, mint=7401msec, maxt=7816msec   *
4/16384 -> WRITE: io=32768MB, aggrb=4172.2MB/s, minb=534034KB/s, 
maxb=581089KB/s, mint=7218msec, maxt=7854msec
5/16384 -> WRITE: io=32768MB, aggrb=4175.4MB/s, minb=534442KB/s, 
maxb=587108KB/s, mint=7144msec, maxt=7848msec
6/16384 -> WRITE: io=32768MB, aggrb=4188.2MB/s, minb=536081KB/s, 
maxb=585224KB/s, mint=7167msec, maxt=7824msec
7/16384 -> WRITE: io=32768MB, aggrb=4173.8MB/s, minb=534238KB/s, 
maxb=591330KB/s, mint=7093msec, maxt=7851msec
8/16384 -> WRITE: io=32768MB, aggrb=4163.2MB/s, minb=532880KB/s, 
maxb=590165KB/s, mint=7107msec, maxt=7871msec
9/16384 -> WRITE: io=32768MB, aggrb=4166.9MB/s, minb=533355KB/s, 
maxb=608664KB/s, mint=6891msec, maxt=7864msec
10/16384 -> WRITE: io=32768MB, aggrb=4157.9MB/s, minb=532204KB/s, 
maxb=594768KB/s, mint=7052msec, maxt=7881msec

0/32768 -> WRITE: io=32768MB, aggrb=1288.1MB/s, minb=164980KB/s, 
maxb=189026KB/s, mint=22189msec, maxt=25423msec
1/32768 -> WRITE: io=32768MB, aggrb=2443.6MB/s, minb=312774KB/s, 
maxb=348624KB/s, mint=12031msec, maxt=13410msec
2/32768 -> WRITE: io=32768MB, aggrb=3467.1MB/s, minb=443888KB/s, 
maxb=484722KB/s, mint=8653msec, maxt=9449msec
3/32768 -> WRITE: io=32768MB, aggrb=4131.2MB/s, minb=528782KB/s, 
maxb=572444KB/s, mint=7327msec, maxt=7932msec    *
4/32768 -> WRITE: io=32768MB, aggrb=4082.8MB/s, minb=522589KB/s, 
maxb=606990KB/s, mint=6910msec, maxt=8026msec
5/32768 -> WRITE: io=32768MB, aggrb=3985.5MB/s, minb=510131KB/s, 
maxb=578046KB/s, mint=7256msec, maxt=8222msec
6/32768 -> WRITE: io=32768MB, aggrb=3937.2MB/s, minb=504062KB/s, 
maxb=591914KB/s, mint=7086msec, maxt=8321msec
7/32768 -> WRITE: io=32768MB, aggrb=4012.3MB/s, minb=513567KB/s, 
maxb=583028KB/s, mint=7194msec, maxt=8167msec
8/32768 -> WRITE: io=32768MB, aggrb=3944.2MB/s, minb=504851KB/s, 
maxb=567257KB/s, mint=7394msec, maxt=8308msec
9/32768 -> WRITE: io=32768MB, aggrb=3930.1MB/s, minb=503155KB/s, 
maxb=580687KB/s, mint=7223msec, maxt=8336msec
10/32768 -> WRITE: io=32768MB, aggrb=3965.2MB/s, minb=507539KB/s, 
maxb=599443KB/s, mint=6997msec, maxt=8264msec


Analysis:
-----------
- the amount of minimum stripe cache entries doesn't cause much 
variation as expected
- writing threads cause significant performance enhancement
- seen best results with 3 or 4 writing threads which correlates well to 
the # of stripes


Did you provide your fio job(s) for comparision yet?

Regards,
Heinz

P.S.: write performance tested with the following script:

#!/bin/sh

MD=md0

for s in 256 512 1024 2048 4096 8192 16384 32768
do
         echo $s > /sys/block/$MD/md/stripe_cache_size

         for t in {0..10}
         do
                 echo $t > /sys/block/$MD/md/group_thread_cnt
                 echo -n "$t/$s -> "
                 fio  --section=w-md0 fio_md0.job 2>&1|grep "aggrb="|sed 
's/^ *//'
         done
done



On 01/17/2017 04:28 PM, Jake Yao wrote:
> Thanks for the response.
>
> I am using fio for performance measurement.
>
> The chunk size of raid5 array is 32K, and the block size in fio is set
> to 96K(3x chunk size) which is also the optimal_io_size, ioengine is
> set to libaio with direct IO.
>
> Increasing stripe_cache_size does not help much, and it looks like the
> write is limited by the single kernel thread as mentioned earlier.
>
>
> On Tue, Jan 17, 2017 at 12:10 AM, Roman Mamedov <rm@romanrm.net> wrote:
>> On Mon, 16 Jan 2017 21:35:21 -0500
>> Jake Yao <jgyao1@gmail.com> wrote:
>>
>>> I have a raid5 array on 4 NVMe drives, and the performance on the
>>> array is only marginally better than a single drive. Unlike a similar
>>> raid5 array on 4 SAS SSD or HDD,  the performance on array is 3x
>>> better than a single drive, which is expected.
>>>
>>> It looks like when the single kernel thread associated with the raid
>>> device running at 100%, the array performance hit its peak. This can
>>> happen easily for fast devices like NVMe.
>>>
>>> This can reproduced by creating a raid5 with 4 ramdisks as well, and
>>> comparing performance on the array and one ramdisk. Sometimes the
>>> performance on the array is worse than a single ramdisk.
>>>
>>> The kernel version is 4.9.0-rc3 and mdadm is release 3.4, no write
>>> journal is configured.
>>>
>>> Is this a known issue?
>> How do you measure the performance?
>>
>> Sure it may be CPU-bound in the end, but also why not try the usual
>> optimization tricks, such as:
>>
>>    * increase your stripe_cache_size, it's not uncommon that this can speed up
>>      linear writes by as much as several times;
>>
>>    * if you meant reads, you could look into read-ahead settings for the array;
>>
>>    * and in both cases, try experimenting with different stripe sizes (if you
>>      were using 512K, try with 64K stripes).
>>
>> --
>> With respect,
>> Roman
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: performance of raid5 on fast devices

[Jake Yao]

It is interesting. I do not see the similar behavior with the change
of group_thread_cnt.

The raid5 I have is following:

md125 : active raid5 nvme0n1p1[0] nvme2n1p1[2] nvme1n1p1[1] nvme3n1p1[4]
      943325184 blocks super 1.2 level 5, 32k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/3 pages [0KB], 65536KB chunk

/dev/md125:
        Version : 1.2
  Creation Time : Thu Dec 15 20:11:46 2016
     Raid Level : raid5
     Array Size : 943325184 (899.63 GiB 965.96 GB)
  Used Dev Size : 314441728 (299.88 GiB 321.99 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Wed Jan 18 16:24:52 2017
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 32K

           Name : localhost:nvme  (local to host localhost)
           UUID : 477a94af:79f5a10a:0d513dc6:7f5e670d
         Events : 108

    Number   Major   Minor   RaidDevice State
       0     259        6        0      active sync   /dev/nvme0n1p1
       1     259        8        1      active sync   /dev/nvme1n1p1
       2     259        9        2      active sync   /dev/nvme2n1p1
       4     259        1        3      active sync   /dev/nvme3n1p1

The fio config is:

[global]
ioengine=libaio
iodepth=64
bs=96K
direct=1
thread=1
time_based=1
runtime=20
numjobs=1
loops=1
group_reporting=1
exitall

[nvme_md_wrt]
rw=write
filename=/dev/md125

[nvme_single_wrt]
rw=write
filename=/dev/nvme1n1p2

With changing group_thread_cnt, I got following:

0 -> WRITE: io=40643MB, aggrb=2031.1MB/s, minb=2031.1MB/s,
maxb=2031.1MB/s, mint=20002msec, maxt=20002msec
1 -> WRITE: io=43740MB, aggrb=2186.7MB/s, minb=2186.7MB/s,
maxb=2186.7MB/s, mint=20003msec, maxt=20003msec
2 -> WRITE: io=43805MB, aggrb=2189.1MB/s, minb=2189.1MB/s,
maxb=2189.1MB/s, mint=20003msec, maxt=20003msec
3 -> WRITE: io=43763MB, aggrb=2187.9MB/s, minb=2187.9MB/s,
maxb=2187.9MB/s, mint=20003msec, maxt=20003msec
4 -> WRITE: io=43767MB, aggrb=2188.2MB/s, minb=2188.2MB/s,
maxb=2188.2MB/s, mint=20002msec, maxt=20002msec
5 -> WRITE: io=43767MB, aggrb=2188.4MB/s, minb=2188.4MB/s,
maxb=2188.4MB/s, mint=20003msec, maxt=20003msec
6 -> WRITE: io=43776MB, aggrb=2188.5MB/s, minb=2188.5MB/s,
maxb=2188.5MB/s, mint=20003msec, maxt=20003msec
7 -> WRITE: io=43758MB, aggrb=2187.6MB/s, minb=2187.6MB/s,
maxb=2187.6MB/s, mint=20003msec, maxt=20003msec
8 -> WRITE: io=43766MB, aggrb=2187.1MB/s, minb=2187.1MB/s,
maxb=2187.1MB/s, mint=20003msec, maxt=20003msec

In the test run,  the md125_raid5 kernel thread running close to 100%
during the test, and all the kworker threads at around 10%

My system is a VM with 6 cpus running on ESXi with NVMe drives passthru.

I am wondering why the difference.

Thanks!


On Tue, Jan 17, 2017 at 4:04 PM, Heinz Mauelshagen <heinzm@redhat.com> wrote:
> Jake et al,
>
> I took the oportunity to measure raid5 on a 4x NVME here with
> variations of group_thread_cnt={0..10} minimal
> stripe_cache_size={256,512,1024,2048,4096,8192,16384,32768}
>
> This is on an X-99 with Intel E5-2640 and kernel 4.9.3-200.fc25.x86_64.
>
> Highest active stripe count logged < 17K.
>
>
> fio job/sections used:
> ----------------------------
> [r-md0]
> ioengine=libaio
> iodepth=40
> rw=read
> bs=4096K
> direct=1
> size=4G
> numjobs=8
> filename=/dev/md0
>
> [w-md0]
> ioengine=libaio
> iodepth=40
> rw=write
> bs=4096K
> direct=1
> size=4G
> numjobs=8
> filename=/dev/md0
>
>
> Baseline performance seen with raid0:
> ---------------------------------------------------
> md0 : active raid0 dm-350[3] dm-349[2] dm-348[1] dm-347[0]
>       33521664 blocks super 1.2 32k chunks
>
> READ: io=32768MB, aggrb=8202.3MB/s, minb=1025.3MB/s, maxb=1217.7MB/s,
> mint=3364msec, maxt=3995msec
> WRITE: io=32768MB, aggrb=5746.8MB/s, minb=735584KB/s, maxb=836685KB/s,
> mint=5013msec, maxt=5702msec
>
>
> Performance with raid5:
> --------------------------------
> md0 : active raid5 dm-350[3] dm-349[2] dm-348[1] dm-347[0]
>       25141248 blocks super 1.2 level 5, 32k chunk, algorithm 2 [4/4] [UUUU]
>
>
> READ: io=32768MB, aggrb=7375.3MB/s, minb=944025KB/s, maxb=1001.1MB/s,
> mint=4088msec, maxt=4443msec
>
>
> Write results for group_thread_cnt/stripe_cache_size variations:
> ------------------------------------------------------------------------------------
> 0/256  -> WRITE: io=32768MB, aggrb=1296.4MB/s, minb=165927KB/s,
> maxb=167644KB/s, mint=25019msec, maxt=25278msec
> 1/256  -> WRITE: io=32768MB, aggrb=2152.6MB/s, minb=275524KB/s,
> maxb=278654KB/s, mint=15052msec, maxt=15223msec
> 2/256  -> WRITE: io=32768MB, aggrb=3177.4MB/s, minb=406700KB/s,
> maxb=415854KB/s, mint=10086msec, maxt=10313msec
> 3/256  -> WRITE: io=32768MB, aggrb=4026.6MB/s, minb=515397KB/s,
> maxb=524222KB/s, mint=8001msec, maxt=8138msec
> 4/256  -> WRITE: io=32768MB, aggrb=4172.2MB/s, minb=534034KB/s,
> maxb=552609KB/s, mint=7590msec, maxt=7854msec  *
> 5/256  -> WRITE: io=32768MB, aggrb=4166.9MB/s, minb=533355KB/s,
> maxb=547845KB/s, mint=7656msec, maxt=7864msec
> 6/256  -> WRITE: io=32768MB, aggrb=4189.3MB/s, minb=536218KB/s,
> maxb=556126KB/s, mint=7542msec, maxt=7822msec
> 7/256  -> WRITE: io=32768MB, aggrb=4192.5MB/s, minb=536630KB/s,
> maxb=560810KB/s, mint=7479msec, maxt=7816msec
> 8/256  -> WRITE: io=32768MB, aggrb=4185.2MB/s, minb=535807KB/s,
> maxb=562389KB/s, mint=7458msec, maxt=7828msec
> 9/256  -> WRITE: io=32768MB, aggrb=4192.1MB/s, minb=536699KB/s,
> maxb=577966KB/s, mint=7257msec, maxt=7815msec
> 10/256 -> WRITE: io=32768MB, aggrb=4182.3MB/s, minb=535329KB/s,
> maxb=568256KB/s, mint=7381msec, maxt=7835msec
>
> 0/512 -> WRITE: io=32768MB, aggrb=1297.8MB/s, minb=166025KB/s,
> maxb=167664KB/s, mint=25016msec, maxt=25263msec
> 1/512 -> WRITE: io=32768MB, aggrb=2148.5MB/s, minb=275000KB/s,
> maxb=278044KB/s, mint=15085msec, maxt=15252msec
> 2/512 -> WRITE: io=32768MB, aggrb=3158.4MB/s, minb=404270KB/s,
> maxb=411407KB/s, mint=10195msec, maxt=10375msec
> 3/512 -> WRITE: io=32768MB, aggrb=4102.7MB/s, minb=525141KB/s,
> maxb=539738KB/s, mint=7771msec, maxt=7987msec
> 4/512 -> WRITE: io=32768MB, aggrb=4162.8MB/s, minb=532745KB/s,
> maxb=541759KB/s, mint=7742msec, maxt=7873msec     *
> 5/512 -> WRITE: io=32768MB, aggrb=4178.6MB/s, minb=534851KB/s,
> maxb=549856KB/s, mint=7628msec, maxt=7842msec
> 6/512 -> WRITE: io=32768MB, aggrb=4167.4MB/s, minb=533422KB/s,
> maxb=562314KB/s, mint=7459msec, maxt=7863msec
> 7/512 -> WRITE: io=32768MB, aggrb=4192.1MB/s, minb=536699KB/s,
> maxb=566338KB/s, mint=7406msec, maxt=7815msec
> 8/512 -> WRITE: io=32768MB, aggrb=4189.8MB/s, minb=536287KB/s,
> maxb=558644KB/s, mint=7508msec, maxt=7821msec
> 9/512 -> WRITE: io=32768MB, aggrb=4165.8MB/s, minb=533219KB/s,
> maxb=559837KB/s, mint=7492msec, maxt=7866msec
> 10/512 -> WRITE: io=32768MB, aggrb=4177.2MB/s, minb=534783KB/s,
> maxb=570188KB/s, mint=7356msec, maxt=7843msec
>
> 0/1024 -> WRITE: io=32768MB, aggrb=1288.6MB/s, minb=164935KB/s,
> maxb=166877KB/s, mint=25134msec, maxt=25430msec
> 1/1024 -> WRITE: io=32768MB, aggrb=2218.5MB/s, minb=283955KB/s,
> maxb=289842KB/s, mint=14471msec, maxt=14771msec
> 2/1024 -> WRITE: io=32768MB, aggrb=3186.1MB/s, minb=407926KB/s,
> maxb=420903KB/s, mint=9965msec, maxt=10282msec
> 3/1024 -> WRITE: io=32768MB, aggrb=4107.4MB/s, minb=525733KB/s,
> maxb=538836KB/s, mint=7784msec, maxt=7978msec
> 4/1024 -> WRITE: io=32768MB, aggrb=4146.9MB/s, minb=530790KB/s,
> maxb=550505KB/s, mint=7619msec, maxt=7902msec
> 5/1024 -> WRITE: io=32768MB, aggrb=4160.5MB/s, minb=532542KB/s,
> maxb=550795KB/s, mint=7615msec, maxt=7876msec  *
> 6/1024 -> WRITE: io=32768MB, aggrb=4174.3MB/s, minb=534306KB/s,
> maxb=558942KB/s, mint=7504msec, maxt=7850msec
> 7/1024 -> WRITE: io=32768MB, aggrb=4189.8MB/s, minb=536287KB/s,
> maxb=556864KB/s, mint=7532msec, maxt=7821msec
> 8/1024 -> WRITE: io=32768MB, aggrb=4188.2MB/s, minb=536081KB/s,
> maxb=561035KB/s, mint=7476msec, maxt=7824msec
> 9/1024 -> WRITE: io=32768MB, aggrb=4167.4MB/s, minb=533422KB/s,
> maxb=567872KB/s, mint=7386msec, maxt=7863msec
> 10/1024 -> WRITE: io=32768MB, aggrb=4188.2MB/s, minb=536081KB/s,
> maxb=569878KB/s, mint=7360msec, maxt=7824msec
>
> 0/2048 -> WRITE: io=32768MB, aggrb=1265.7MB/s, minb=162004KB/s,
> maxb=166111KB/s, mint=25250msec, maxt=25890msec
> 1/2048 -> WRITE: io=32768MB, aggrb=2239.5MB/s, minb=286652KB/s,
> maxb=290846KB/s, mint=14421msec, maxt=14632msec
> 2/2048 -> WRITE: io=32768MB, aggrb=3184.5MB/s, minb=407609KB/s,
> maxb=413150KB/s, mint=10152msec, maxt=10290msec
> 3/2048 -> WRITE: io=32768MB, aggrb=4213.5MB/s, minb=539321KB/s,
> maxb=557901KB/s, mint=7518msec, maxt=7777msec     *
> 4/2048 -> WRITE: io=32768MB, aggrb=4168.5MB/s, minb=533558KB/s,
> maxb=543162KB/s, mint=7722msec, maxt=7861msec
> 5/2048 -> WRITE: io=32768MB, aggrb=4185.5MB/s, minb=535739KB/s,
> maxb=549352KB/s, mint=7635msec, maxt=7829msec
> 6/2048 -> WRITE: io=32768MB, aggrb=4181.8MB/s, minb=535260KB/s,
> maxb=553338KB/s, mint=7580msec, maxt=7836msec
> 7/2048 -> WRITE: io=32768MB, aggrb=4215.7MB/s, minb=539599KB/s,
> maxb=566109KB/s, mint=7409msec, maxt=7773msec
> 8/2048 -> WRITE: io=32768MB, aggrb=4200.5MB/s, minb=537662KB/s,
> maxb=568102KB/s, mint=7383msec, maxt=7801msec
> 9/2048 -> WRITE: io=32768MB, aggrb=4184.1MB/s, minb=535671KB/s,
> maxb=574483KB/s, mint=7301msec, maxt=7830msec
> 10/2048 -> WRITE: io=32768MB, aggrb=4172.7MB/s, minb=534102KB/s,
> maxb=567641KB/s, mint=7389msec, maxt=7853msec
>
> 0/4096 -> WRITE: io=32768MB, aggrb=1264.8MB/s, minb=161879KB/s,
> maxb=168588KB/s, mint=24879msec, maxt=25910msec
> 1/4096 -> WRITE: io=32768MB, aggrb=2349.4MB/s, minb=300710KB/s,
> maxb=312541KB/s, mint=13420msec, maxt=13948msec
> 2/4096 -> WRITE: io=32768MB, aggrb=3387.6MB/s, minb=433609KB/s,
> maxb=441877KB/s, mint=9492msec, maxt=9673msec
> 3/4096 -> WRITE: io=32768MB, aggrb=4182.3MB/s, minb=535329KB/s,
> maxb=552390KB/s, mint=7593msec, maxt=7835msec    *
> 4/4096 -> WRITE: io=32768MB, aggrb=4170.2MB/s, minb=533762KB/s,
> maxb=560061KB/s, mint=7489msec, maxt=7858msec
> 5/4096 -> WRITE: io=32768MB, aggrb=4179.6MB/s, minb=534919KB/s,
> maxb=548490KB/s, mint=7647msec, maxt=7841msec
> 6/4096 -> WRITE: io=32768MB, aggrb=4183.4MB/s, minb=535465KB/s,
> maxb=549208KB/s, mint=7637msec, maxt=7833msec
> 7/4096 -> WRITE: io=32768MB, aggrb=4174.9MB/s, minb=534374KB/s,
> maxb=557530KB/s, mint=7523msec, maxt=7849msec
> 8/4096 -> WRITE: io=32768MB, aggrb=4178.6MB/s, minb=534851KB/s,
> maxb=570188KB/s, mint=7356msec, maxt=7842msec
> 9/4096 -> WRITE: io=32768MB, aggrb=4180.2MB/s, minb=535056KB/s,
> maxb=570110KB/s, mint=7357msec, maxt=7839msec
> 10/4096 -> WRITE: io=32768MB, aggrb=4183.9MB/s, minb=535534KB/s,
> maxb=574640KB/s, mint=7299msec, maxt=7832msec
>
> 0/8192 -> WRITE: io=32768MB, aggrb=1260.9MB/s, minb=161381KB/s,
> maxb=171511KB/s, mint=24455msec, maxt=25990msec
> 1/8192 -> WRITE: io=32768MB, aggrb=2368.5MB/s, minb=303166KB/s,
> maxb=320444KB/s, mint=13089msec, maxt=13835msec
> 2/8192 -> WRITE: io=32768MB, aggrb=3408.8MB/s, minb=436225KB/s,
> maxb=458544KB/s, mint=9147msec, maxt=9615msec
> 3/8192 -> WRITE: io=32768MB, aggrb=4219.5MB/s, minb=540085KB/s,
> maxb=564585KB/s, mint=7429msec, maxt=7766msec     *
> 4/8192 -> WRITE: io=32768MB, aggrb=4208.6MB/s, minb=538698KB/s,
> maxb=570653KB/s, mint=7350msec, maxt=7786msec
> 5/8192 -> WRITE: io=32768MB, aggrb=4200.5MB/s, minb=537662KB/s,
> maxb=562013KB/s, mint=7463msec, maxt=7801msec
> 6/8192 -> WRITE: io=32768MB, aggrb=4189.3MB/s, minb=536218KB/s,
> maxb=585387KB/s, mint=7165msec, maxt=7822msec
> 7/8192 -> WRITE: io=32768MB, aggrb=4184.5MB/s, minb=535602KB/s,
> maxb=579323KB/s, mint=7240msec, maxt=7831msec
> 8/8192 -> WRITE: io=32768MB, aggrb=4186.6MB/s, minb=535876KB/s,
> maxb=572132KB/s, mint=7331msec, maxt=7827msec
> 9/8192 -> WRITE: io=32768MB, aggrb=4176.5MB/s, minb=534578KB/s,
> maxb=598246KB/s, mint=7011msec, maxt=7846msec
> 10/8192 -> WRITE: io=32768MB, aggrb=4184.1MB/s, minb=535671KB/s,
> maxb=580285KB/s, mint=7228msec, maxt=7830msec
>
> 0/16384 -> WRITE: io=32768MB, aggrb=1281.0MB/s, minb=163968KB/s,
> maxb=183542KB/s, mint=22852msec, maxt=25580msec
> 1/16384 -> WRITE: io=32768MB, aggrb=2451.8MB/s, minb=313827KB/s,
> maxb=337787KB/s, mint=12417msec, maxt=13365msec
> 2/16384 -> WRITE: io=32768MB, aggrb=3409.5MB/s, minb=436406KB/s,
> maxb=468532KB/s, mint=8952msec, maxt=9611msec
> 3/16384 -> WRITE: io=32768MB, aggrb=4192.5MB/s, minb=536630KB/s,
> maxb=566721KB/s, mint=7401msec, maxt=7816msec   *
> 4/16384 -> WRITE: io=32768MB, aggrb=4172.2MB/s, minb=534034KB/s,
> maxb=581089KB/s, mint=7218msec, maxt=7854msec
> 5/16384 -> WRITE: io=32768MB, aggrb=4175.4MB/s, minb=534442KB/s,
> maxb=587108KB/s, mint=7144msec, maxt=7848msec
> 6/16384 -> WRITE: io=32768MB, aggrb=4188.2MB/s, minb=536081KB/s,
> maxb=585224KB/s, mint=7167msec, maxt=7824msec
> 7/16384 -> WRITE: io=32768MB, aggrb=4173.8MB/s, minb=534238KB/s,
> maxb=591330KB/s, mint=7093msec, maxt=7851msec
> 8/16384 -> WRITE: io=32768MB, aggrb=4163.2MB/s, minb=532880KB/s,
> maxb=590165KB/s, mint=7107msec, maxt=7871msec
> 9/16384 -> WRITE: io=32768MB, aggrb=4166.9MB/s, minb=533355KB/s,
> maxb=608664KB/s, mint=6891msec, maxt=7864msec
> 10/16384 -> WRITE: io=32768MB, aggrb=4157.9MB/s, minb=532204KB/s,
> maxb=594768KB/s, mint=7052msec, maxt=7881msec
>
> 0/32768 -> WRITE: io=32768MB, aggrb=1288.1MB/s, minb=164980KB/s,
> maxb=189026KB/s, mint=22189msec, maxt=25423msec
> 1/32768 -> WRITE: io=32768MB, aggrb=2443.6MB/s, minb=312774KB/s,
> maxb=348624KB/s, mint=12031msec, maxt=13410msec
> 2/32768 -> WRITE: io=32768MB, aggrb=3467.1MB/s, minb=443888KB/s,
> maxb=484722KB/s, mint=8653msec, maxt=9449msec
> 3/32768 -> WRITE: io=32768MB, aggrb=4131.2MB/s, minb=528782KB/s,
> maxb=572444KB/s, mint=7327msec, maxt=7932msec    *
> 4/32768 -> WRITE: io=32768MB, aggrb=4082.8MB/s, minb=522589KB/s,
> maxb=606990KB/s, mint=6910msec, maxt=8026msec
> 5/32768 -> WRITE: io=32768MB, aggrb=3985.5MB/s, minb=510131KB/s,
> maxb=578046KB/s, mint=7256msec, maxt=8222msec
> 6/32768 -> WRITE: io=32768MB, aggrb=3937.2MB/s, minb=504062KB/s,
> maxb=591914KB/s, mint=7086msec, maxt=8321msec
> 7/32768 -> WRITE: io=32768MB, aggrb=4012.3MB/s, minb=513567KB/s,
> maxb=583028KB/s, mint=7194msec, maxt=8167msec
> 8/32768 -> WRITE: io=32768MB, aggrb=3944.2MB/s, minb=504851KB/s,
> maxb=567257KB/s, mint=7394msec, maxt=8308msec
> 9/32768 -> WRITE: io=32768MB, aggrb=3930.1MB/s, minb=503155KB/s,
> maxb=580687KB/s, mint=7223msec, maxt=8336msec
> 10/32768 -> WRITE: io=32768MB, aggrb=3965.2MB/s, minb=507539KB/s,
> maxb=599443KB/s, mint=6997msec, maxt=8264msec
>
>
> Analysis:
> -----------
> - the amount of minimum stripe cache entries doesn't cause much variation as
> expected
> - writing threads cause significant performance enhancement
> - seen best results with 3 or 4 writing threads which correlates well to the
> # of stripes
>
>
> Did you provide your fio job(s) for comparision yet?
>
> Regards,
> Heinz
>
> P.S.: write performance tested with the following script:
>
> #!/bin/sh
>
> MD=md0
>
> for s in 256 512 1024 2048 4096 8192 16384 32768
> do
>         echo $s > /sys/block/$MD/md/stripe_cache_size
>
>         for t in {0..10}
>         do
>                 echo $t > /sys/block/$MD/md/group_thread_cnt
>                 echo -n "$t/$s -> "
>                 fio  --section=w-md0 fio_md0.job 2>&1|grep "aggrb="|sed 's/^
> *//'
>         done
> done
>
>
>
>
> On 01/17/2017 04:28 PM, Jake Yao wrote:
>>
>> Thanks for the response.
>>
>> I am using fio for performance measurement.
>>
>> The chunk size of raid5 array is 32K, and the block size in fio is set
>> to 96K(3x chunk size) which is also the optimal_io_size, ioengine is
>> set to libaio with direct IO.
>>
>> Increasing stripe_cache_size does not help much, and it looks like the
>> write is limited by the single kernel thread as mentioned earlier.
>>
>>
>> On Tue, Jan 17, 2017 at 12:10 AM, Roman Mamedov <rm@romanrm.net> wrote:
>>>
>>> On Mon, 16 Jan 2017 21:35:21 -0500
>>> Jake Yao <jgyao1@gmail.com> wrote:
>>>
>>>> I have a raid5 array on 4 NVMe drives, and the performance on the
>>>> array is only marginally better than a single drive. Unlike a similar
>>>> raid5 array on 4 SAS SSD or HDD,  the performance on array is 3x
>>>> better than a single drive, which is expected.
>>>>
>>>> It looks like when the single kernel thread associated with the raid
>>>> device running at 100%, the array performance hit its peak. This can
>>>> happen easily for fast devices like NVMe.
>>>>
>>>> This can reproduced by creating a raid5 with 4 ramdisks as well, and
>>>> comparing performance on the array and one ramdisk. Sometimes the
>>>> performance on the array is worse than a single ramdisk.
>>>>
>>>> The kernel version is 4.9.0-rc3 and mdadm is release 3.4, no write
>>>> journal is configured.
>>>>
>>>> Is this a known issue?
>>>
>>> How do you measure the performance?
>>>
>>> Sure it may be CPU-bound in the end, but also why not try the usual
>>> optimization tricks, such as:
>>>
>>>    * increase your stripe_cache_size, it's not uncommon that this can
>>> speed up
>>>      linear writes by as much as several times;
>>>
>>>    * if you meant reads, you could look into read-ahead settings for the
>>> array;
>>>
>>>    * and in both cases, try experimenting with different stripe sizes (if
>>> you
>>>      were using 512K, try with 64K stripes).
>>>
>>> --
>>> With respect,
>>> Roman
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: performance of raid5 on fast devices

[Coly Li]

On 2017/1/19 上午3:25, Jake Yao wrote:
> It is interesting. I do not see the similar behavior with the change
> of group_thread_cnt.
> 
> The raid5 I have is following:
> 
> md125 : active raid5 nvme0n1p1[0] nvme2n1p1[2] nvme1n1p1[1] nvme3n1p1[4]
>       943325184 blocks super 1.2 level 5, 32k chunk, algorithm 2 [4/4] [UUUU]
>       bitmap: 0/3 pages [0KB], 65536KB chunk
> 
> /dev/md125:
>         Version : 1.2
>   Creation Time : Thu Dec 15 20:11:46 2016
>      Raid Level : raid5
>      Array Size : 943325184 (899.63 GiB 965.96 GB)
>   Used Dev Size : 314441728 (299.88 GiB 321.99 GB)
>    Raid Devices : 4
>   Total Devices : 4
>     Persistence : Superblock is persistent
> 
>   Intent Bitmap : Internal
> 
>     Update Time : Wed Jan 18 16:24:52 2017
>           State : clean
>  Active Devices : 4
> Working Devices : 4
>  Failed Devices : 0
>   Spare Devices : 0
> 
>          Layout : left-symmetric
>      Chunk Size : 32K
> 
>            Name : localhost:nvme  (local to host localhost)
>            UUID : 477a94af:79f5a10a:0d513dc6:7f5e670d
>          Events : 108
> 
>     Number   Major   Minor   RaidDevice State
>        0     259        6        0      active sync   /dev/nvme0n1p1
>        1     259        8        1      active sync   /dev/nvme1n1p1
>        2     259        9        2      active sync   /dev/nvme2n1p1
>        4     259        1        3      active sync   /dev/nvme3n1p1
> 
> The fio config is:
> 
> [global]
> ioengine=libaio
> iodepth=64
> bs=96K
> direct=1
> thread=1
> time_based=1
> runtime=20
> numjobs=1

You only have 1 I/O thread, bottle neck is here. Have a try with numjobs=8.

> loops=1
> group_reporting=1
> exitall
[snip]

Coly

Re: performance of raid5 on fast devices

[Jake Yao]

I run tests with multiple IO threads, but it looks like it does not
affect the overall performance.

In this run with 8 io threads,

[global]
ioengine=libaio
iodepth=64
bs=192k
direct=1
thread=1
time_based=1
runtime=20
numjobs=8
loops=1
group_reporting=1
rwmixread=70
rwmixwrite=30
exitall
#
# end of global
#
[nvme_md_write]
rw=write
filename=/dev/md127
runtime=20

[nvme_drv_write]
rw=write
filename=/dev/nvme1n1p2
runtime=20

I got following for nvme based raid5 and single drive:

md thrd-cnt 0: write: io=27992MB, bw=1397.5MB/s, iops=7452, runt= 20031msec
md thrd-cnt 1: write: io=43065MB, bw=2148.6MB/s, iops=11458, runt= 20044msec
md thrd-cnt 2: write: io=43209MB, bw=2155.9MB/s, iops=11497, runt= 20043msec
md thrd-cnt 3: write: io=43163MB, bw=2153.9MB/s, iops=11487, runt= 20040msec
md thrd-cnt 4: write: io=43316MB, bw=2163.2MB/s, iops=11536, runt= 20024msec
md thrd-cnt 5: write: io=43390MB, bw=2164.7MB/s, iops=11544, runt= 20045msec
md thrd-cnt 6: write: io=43295MB, bw=2160.2MB/s, iops=11521, runt= 20042msec
single drive: write: io=36004MB, bw=1795.4MB/s, iops=9575, runt= 20054msec

It also does not show little effect on ssd based raid5 and single
drive. Same fio config as above, just changing the corresponding
device filenames. The result is following:

md thrd-cnt 0: write: io=13646MB, bw=696242KB/s, iops=3626, runt= 20070msec
md thrd-cnt 1: write: io=24519MB, bw=1221.5MB/s, iops=6514, runt= 20074msec
md thrd-cnt 2: write: io=24780MB, bw=1234.9MB/s, iops=6585, runt= 20068msec
md thrd-cnt 3: write: io=24890MB, bw=1240.2MB/s, iops=6613, runt= 20072msec
md thrd-cnt 4: write: io=24937MB, bw=1242.5MB/s, iops=6626, runt= 20071msec
md thrd-cnt 5: write: io=24948MB, bw=1242.9MB/s, iops=6628, runt= 20073msec
md thrd-cnt 6: write: io=24701MB, bw=1230.1MB/s, iops=6564, runt= 20068msec
single drive: write: io=8389.4MB, bw=428184KB/s, iops=2230, runt= 20063msec

In the ssd case, raid5 array is 3x better than a single drive.

On Fri, Jan 20, 2017 at 9:58 AM, Coly Li <colyli@suse.de> wrote:
> On 2017/1/19 上午3:25, Jake Yao wrote:
>> It is interesting. I do not see the similar behavior with the change
>> of group_thread_cnt.
>>
>> The raid5 I have is following:
>>
>> md125 : active raid5 nvme0n1p1[0] nvme2n1p1[2] nvme1n1p1[1] nvme3n1p1[4]
>>       943325184 blocks super 1.2 level 5, 32k chunk, algorithm 2 [4/4] [UUUU]
>>       bitmap: 0/3 pages [0KB], 65536KB chunk
>>
>> /dev/md125:
>>         Version : 1.2
>>   Creation Time : Thu Dec 15 20:11:46 2016
>>      Raid Level : raid5
>>      Array Size : 943325184 (899.63 GiB 965.96 GB)
>>   Used Dev Size : 314441728 (299.88 GiB 321.99 GB)
>>    Raid Devices : 4
>>   Total Devices : 4
>>     Persistence : Superblock is persistent
>>
>>   Intent Bitmap : Internal
>>
>>     Update Time : Wed Jan 18 16:24:52 2017
>>           State : clean
>>  Active Devices : 4
>> Working Devices : 4
>>  Failed Devices : 0
>>   Spare Devices : 0
>>
>>          Layout : left-symmetric
>>      Chunk Size : 32K
>>
>>            Name : localhost:nvme  (local to host localhost)
>>            UUID : 477a94af:79f5a10a:0d513dc6:7f5e670d
>>          Events : 108
>>
>>     Number   Major   Minor   RaidDevice State
>>        0     259        6        0      active sync   /dev/nvme0n1p1
>>        1     259        8        1      active sync   /dev/nvme1n1p1
>>        2     259        9        2      active sync   /dev/nvme2n1p1
>>        4     259        1        3      active sync   /dev/nvme3n1p1
>>
>> The fio config is:
>>
>> [global]
>> ioengine=libaio
>> iodepth=64
>> bs=96K
>> direct=1
>> thread=1
>> time_based=1
>> runtime=20
>> numjobs=1
>
> You only have 1 I/O thread, bottle neck is here. Have a try with numjobs=8.
>
>> loops=1
>> group_reporting=1
>> exitall
> [snip]
>
> Coly

Re: performance of raid5 on fast devices

[Coly Li]

Hi Jake,

Hmm, is the hardware powerful enough ? When I did similar testing, I
used a machine with 2x10 core XEON CPU, and 80GB memory.
And could you please try bs=64K? I got a good performance number with
64KB blocksize.

And could you have a look at top out put, are all the CPUs 100%
utilized, or still idle on some CPUs ?

Coly

On 2017/1/24 上午6:20, Jake Yao wrote:
> I run tests with multiple IO threads, but it looks like it does not
> affect the overall performance.
> 
> In this run with 8 io threads,
> 
> [global]
> ioengine=libaio
> iodepth=64
> bs=192k
> direct=1
> thread=1
> time_based=1
> runtime=20
> numjobs=8
> loops=1
> group_reporting=1
> rwmixread=70
> rwmixwrite=30
> exitall
> #
> # end of global
> #
> [nvme_md_write]
> rw=write
> filename=/dev/md127
> runtime=20
> 
> [nvme_drv_write]
> rw=write
> filename=/dev/nvme1n1p2
> runtime=20
> 
> I got following for nvme based raid5 and single drive:
> 
> md thrd-cnt 0: write: io=27992MB, bw=1397.5MB/s, iops=7452, runt= 20031msec
> md thrd-cnt 1: write: io=43065MB, bw=2148.6MB/s, iops=11458, runt= 20044msec
> md thrd-cnt 2: write: io=43209MB, bw=2155.9MB/s, iops=11497, runt= 20043msec
> md thrd-cnt 3: write: io=43163MB, bw=2153.9MB/s, iops=11487, runt= 20040msec
> md thrd-cnt 4: write: io=43316MB, bw=2163.2MB/s, iops=11536, runt= 20024msec
> md thrd-cnt 5: write: io=43390MB, bw=2164.7MB/s, iops=11544, runt= 20045msec
> md thrd-cnt 6: write: io=43295MB, bw=2160.2MB/s, iops=11521, runt= 20042msec
> single drive: write: io=36004MB, bw=1795.4MB/s, iops=9575, runt= 20054msec
> 
> It also does not show little effect on ssd based raid5 and single
> drive. Same fio config as above, just changing the corresponding
> device filenames. The result is following:
> 
> md thrd-cnt 0: write: io=13646MB, bw=696242KB/s, iops=3626, runt= 20070msec
> md thrd-cnt 1: write: io=24519MB, bw=1221.5MB/s, iops=6514, runt= 20074msec
> md thrd-cnt 2: write: io=24780MB, bw=1234.9MB/s, iops=6585, runt= 20068msec
> md thrd-cnt 3: write: io=24890MB, bw=1240.2MB/s, iops=6613, runt= 20072msec
> md thrd-cnt 4: write: io=24937MB, bw=1242.5MB/s, iops=6626, runt= 20071msec
> md thrd-cnt 5: write: io=24948MB, bw=1242.9MB/s, iops=6628, runt= 20073msec
> md thrd-cnt 6: write: io=24701MB, bw=1230.1MB/s, iops=6564, runt= 20068msec
> single drive: write: io=8389.4MB, bw=428184KB/s, iops=2230, runt= 20063msec
> 
> In the ssd case, raid5 array is 3x better than a single drive.
> 
> On Fri, Jan 20, 2017 at 9:58 AM, Coly Li <colyli@suse.de> wrote:
>> On 2017/1/19 上午3:25, Jake Yao wrote:
>>> It is interesting. I do not see the similar behavior with the change
>>> of group_thread_cnt.
>>>
>>> The raid5 I have is following:
>>>
>>> md125 : active raid5 nvme0n1p1[0] nvme2n1p1[2] nvme1n1p1[1] nvme3n1p1[4]
>>>       943325184 blocks super 1.2 level 5, 32k chunk, algorithm 2 [4/4] [UUUU]
>>>       bitmap: 0/3 pages [0KB], 65536KB chunk
>>>
>>> /dev/md125:
>>>         Version : 1.2
>>>   Creation Time : Thu Dec 15 20:11:46 2016
>>>      Raid Level : raid5
>>>      Array Size : 943325184 (899.63 GiB 965.96 GB)
>>>   Used Dev Size : 314441728 (299.88 GiB 321.99 GB)
>>>    Raid Devices : 4
>>>   Total Devices : 4
>>>     Persistence : Superblock is persistent
>>>
>>>   Intent Bitmap : Internal
>>>
>>>     Update Time : Wed Jan 18 16:24:52 2017
>>>           State : clean
>>>  Active Devices : 4
>>> Working Devices : 4
>>>  Failed Devices : 0
>>>   Spare Devices : 0
>>>
>>>          Layout : left-symmetric
>>>      Chunk Size : 32K
>>>
>>>            Name : localhost:nvme  (local to host localhost)
>>>            UUID : 477a94af:79f5a10a:0d513dc6:7f5e670d
>>>          Events : 108
>>>
>>>     Number   Major   Minor   RaidDevice State
>>>        0     259        6        0      active sync   /dev/nvme0n1p1
>>>        1     259        8        1      active sync   /dev/nvme1n1p1
>>>        2     259        9        2      active sync   /dev/nvme2n1p1
>>>        4     259        1        3      active sync   /dev/nvme3n1p1
>>>
>>> The fio config is:
>>>
>>> [global]
>>> ioengine=libaio
>>> iodepth=64
>>> bs=96K
>>> direct=1
>>> thread=1
>>> time_based=1
>>> runtime=20
>>> numjobs=1
>>
>> You only have 1 I/O thread, bottle neck is here. Have a try with numjobs=8.
>>
>>> loops=1
>>> group_reporting=1
>>> exitall
>> [snip]
>>
>> Coly