From: George Shuklin <george.shuklin@gmail.com>
To: linux-raid@vger.kernel.org
Subject: md: Speed shrinks with drives number
Date: Thu, 28 Jan 2021 13:20:35 +0200 [thread overview]
Message-ID: <d5acde4e-e3a1-debe-0b8c-866a8cb11584@gmail.com> (raw)
TL;DR; The more drives are in the array, the slower it is.
I found this when I was debugging NVME array issues, but I was able to
reduce it to the case with with ram-disks, so no special hardware involved.
It can be reproduced on any machine with reasonable amount of memory
(16-32Gb). I checked it on Ubuntu Focal (5.4.0-65-generic), Groovy
(5.8.0-41-generic), Centos 8
For a very fast underlying device (NVME or brd: block ram disk) the more
drives are added to the array, the slower it become, and it's irrelevant
of the array type: raid1, 0, 10.
The speed lost is very high: a single ram-disk is yielding 200-250
kIOPS, the raid0 of 4 ram disks yields less than 150kIOPS, and 100 RAM
disks in raid0 yields mere 30 kIOPS.
The script to reproduce the issue:
export NUM=100 # or any other number between 1 and 1920.
modprobe brd rd_nr=${NUM} rd_size=10000
mdadm --create /dev/md42 -n ${NUM} -l raid0 --assume-clean /dev/ram* -e
1.2 --force
fio --name test --ioengine=libaio --blocksize=4k --iodepth=${NUM}
--rw=randwrite --time_based --runtime=5s --ramp_time=1s --fsync=1
--direct=1 --disk_util=0 --filename=/dev/md42
mdadm --stop /dev/md42
rmmod brd
My laptop (baseline for ram0 without raid - 197k IOPS)
disks in array - IOPS
1 - 108k
2 - 103k
4 - 99.0k
8 - 89.8k
16 - 75.5k
32 - 52.6k
64 - 34.7k
128 - 20.5k
256 - 11.3k
512 - 5.9k
1024 - 3.4k
1920 - 1.5k
(without --assume-clean it's the same, using --iodepth=1 intead of
${NUM} gives the same).
I feel that having 1.5k IOPS on RAID0 consisting of 1920 ram-disks is a
bit slow.
P.S. The real issue is with NVME devices in 10 drives per array, they
shows even faster decline, but ram-disk make it more visual.
reply other threads:[~2021-01-28 11:21 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d5acde4e-e3a1-debe-0b8c-866a8cb11584@gmail.com \
--to=george.shuklin@gmail.com \
--cc=linux-raid@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).