Linux-Raid Archives on
 help / color / Atom feed
* md: Speed shrinks with drives number
@ 2021-01-28 11:20 George Shuklin
  0 siblings, 0 replies; only message in thread
From: George Shuklin @ 2021-01-28 11:20 UTC (permalink / raw)
  To: linux-raid

TL;DR; The more drives are in the array, the slower it is.

I found this when I was debugging NVME array issues, but I was able to 
reduce it to the case with with ram-disks, so no special hardware involved.

It can be reproduced on any machine with reasonable amount of memory 
(16-32Gb). I checked it on Ubuntu Focal (5.4.0-65-generic), Groovy 
(5.8.0-41-generic), Centos 8

For a very fast underlying device (NVME or brd: block ram disk) the more 
drives are added to the array, the slower it become, and it's irrelevant 
of the array type: raid1, 0, 10.

The speed lost is very high: a single ram-disk is yielding 200-250 
kIOPS, the raid0 of 4 ram disks yields less than 150kIOPS, and 100 RAM 
disks in raid0 yields mere 30 kIOPS.

The script to reproduce the issue:

export NUM=100  # or any other number between 1 and 1920.

modprobe brd rd_nr=${NUM} rd_size=10000
mdadm --create /dev/md42 -n ${NUM} -l raid0 --assume-clean /dev/ram* -e 
1.2 --force
fio --name test --ioengine=libaio   --blocksize=4k --iodepth=${NUM} 
--rw=randwrite --time_based   --runtime=5s --ramp_time=1s   --fsync=1 
--direct=1 --disk_util=0 --filename=/dev/md42
mdadm --stop /dev/md42
rmmod brd

My laptop (baseline for ram0 without raid - 197k IOPS)

disks in array - IOPS

1 - 108k
2 - 103k
4 - 99.0k
8 - 89.8k
16 - 75.5k
32 - 52.6k
64 - 34.7k
128 - 20.5k
256 - 11.3k
512 - 5.9k
1024 - 3.4k
1920 - 1.5k

(without --assume-clean it's the same, using --iodepth=1 intead of 
${NUM} gives the same).

I feel that having 1.5k IOPS on RAID0 consisting of 1920 ram-disks is a 
bit slow.

P.S. The real issue is with NVME devices in 10 drives per array, they 
shows even faster decline, but ram-disk make it more visual.

^ permalink raw reply	[flat|nested] only message in thread

only message in thread, back to index

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-28 11:20 md: Speed shrinks with drives number George Shuklin

Linux-Raid Archives on

Archives are clonable:
	git clone --mirror linux-raid/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-raid linux-raid/ \
	public-inbox-index linux-raid

Example config snippet for mirrors

Newsgroup available over NNTP:

AGPL code for this site: git clone