Linux-Raid Archives on lore.kernel.org
 help / color / Atom feed
* Re: Regarding Raid-6 array
       [not found] <CAPgOLid8qb3igOttaZx1dSwPRpvHDaFbiqn+mFAaYZDaepijag@mail.gmail.com>
@ 2021-02-01 22:00 ` Song Liu
  0 siblings, 0 replies; only message in thread
From: Song Liu @ 2021-02-01 22:00 UTC (permalink / raw)
  To: Vishal Verma; +Cc: linux-raid


CC the list

Hi Vishal,

> On Jan 27, 2021, at 9:04 PM, Vishal Verma <vverma@digitalocean.com> wrote:
> 
> Hello Song,
> 
> This is Vishal Verma, Performance Engineer at DigitalOcean. 
> 
> I was recently playing with our 6x nvme drive based RAID stack assessing its performance against a RAID-10 array.
> I understand with RAID-10 there is no striping or parity work so its write performance looks really nice.
> 
> But, I was not sure about the RAID-6 piece, specifically the RMW piece. 
> I was running FIO 128K 100% sequential workload to the raid6 drive array with O_DIRECT and noticed how drives performed:
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            0.18    0.00    2.36    0.00    0.00   97.46
> 
> Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
> nvme1n1        546.00 8330.00  69768.00 994900.00 16896.00 240311.00  96.87  96.65    0.38    0.56   0.00   127.78   119.44   0.10  84.60
> nvme0n1        513.00 8333.50  65544.00 982314.00 15873.00 237245.00  96.87  96.61    0.35    0.32   0.00   127.77   117.88   0.10  86.80
> nvme4n1        480.00 8795.50  61440.00 1045290.00 14880.00 252535.00  96.88  96.63    0.35    0.28   0.00   128.00   118.84   0.09  87.00
> nvme3n1        513.00 8425.50  65664.00 1012554.00 15903.00 244868.50  96.88  96.67    0.37    0.70   0.01   128.00   120.18   0.09  84.20
> nvme5n1        497.00 8618.00  63496.00 1011158.00 15377.00 244201.50  96.87  96.59    0.36    0.64   0.00   127.76   117.33   0.10  88.20
> nvme2n1        529.00 8306.50  67712.00 998920.00 16399.00 241243.50  96.88  96.67    0.39    0.37   0.00   128.00   120.26   0.09  83.60
> 
> 
> I was surprised to see the amount of reads the drives were doing even though it was a full 100% write test.
> I understand for every write IO RAID-6 md array has to first read P,Q read the data and then calculate new P and Q and write the data.
> 
> However, do we expect the drives to read that much i.e ~65MB/s (6%) worth of reads for every 1GB/s (100%) of writes?

6% of read for sequential write is not surprising. In the worst case, 
for 4kB write from upper layer, RAID-6 does introduce 2x reads and 3x 
writes: read P, readQ, write data, write P, write Q. The raid layer
will optimize full stripe writes to avoid reads, but it is common for 
upper layer to do non-full-stripe writes. 

Thanks,
Song

[...]

^ permalink raw reply	[flat|nested] only message in thread

only message in thread, back to index

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <CAPgOLid8qb3igOttaZx1dSwPRpvHDaFbiqn+mFAaYZDaepijag@mail.gmail.com>
2021-02-01 22:00 ` Regarding Raid-6 array Song Liu

Linux-Raid Archives on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-raid/0 linux-raid/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-raid linux-raid/ https://lore.kernel.org/linux-raid \
		linux-raid@vger.kernel.org
	public-inbox-index linux-raid

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-raid


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git