* Re: Regarding Raid-6 array
[not found] <CAPgOLid8qb3igOttaZx1dSwPRpvHDaFbiqn+mFAaYZDaepijag@mail.gmail.com>
@ 2021-02-01 22:00 ` Song Liu
0 siblings, 0 replies; only message in thread
From: Song Liu @ 2021-02-01 22:00 UTC (permalink / raw)
To: Vishal Verma; +Cc: linux-raid
CC the list
Hi Vishal,
> On Jan 27, 2021, at 9:04 PM, Vishal Verma <vverma@digitalocean.com> wrote:
>
> Hello Song,
>
> This is Vishal Verma, Performance Engineer at DigitalOcean.
>
> I was recently playing with our 6x nvme drive based RAID stack assessing its performance against a RAID-10 array.
> I understand with RAID-10 there is no striping or parity work so its write performance looks really nice.
>
> But, I was not sure about the RAID-6 piece, specifically the RMW piece.
> I was running FIO 128K 100% sequential workload to the raid6 drive array with O_DIRECT and noticed how drives performed:
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.18 0.00 2.36 0.00 0.00 97.46
>
> Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
> nvme1n1 546.00 8330.00 69768.00 994900.00 16896.00 240311.00 96.87 96.65 0.38 0.56 0.00 127.78 119.44 0.10 84.60
> nvme0n1 513.00 8333.50 65544.00 982314.00 15873.00 237245.00 96.87 96.61 0.35 0.32 0.00 127.77 117.88 0.10 86.80
> nvme4n1 480.00 8795.50 61440.00 1045290.00 14880.00 252535.00 96.88 96.63 0.35 0.28 0.00 128.00 118.84 0.09 87.00
> nvme3n1 513.00 8425.50 65664.00 1012554.00 15903.00 244868.50 96.88 96.67 0.37 0.70 0.01 128.00 120.18 0.09 84.20
> nvme5n1 497.00 8618.00 63496.00 1011158.00 15377.00 244201.50 96.87 96.59 0.36 0.64 0.00 127.76 117.33 0.10 88.20
> nvme2n1 529.00 8306.50 67712.00 998920.00 16399.00 241243.50 96.88 96.67 0.39 0.37 0.00 128.00 120.26 0.09 83.60
>
>
> I was surprised to see the amount of reads the drives were doing even though it was a full 100% write test.
> I understand for every write IO RAID-6 md array has to first read P,Q read the data and then calculate new P and Q and write the data.
>
> However, do we expect the drives to read that much i.e ~65MB/s (6%) worth of reads for every 1GB/s (100%) of writes?
6% of read for sequential write is not surprising. In the worst case,
for 4kB write from upper layer, RAID-6 does introduce 2x reads and 3x
writes: read P, readQ, write data, write P, write Q. The raid layer
will optimize full stripe writes to avoid reads, but it is common for
upper layer to do non-full-stripe writes.
Thanks,
Song
[...]
^ permalink raw reply [flat|nested] only message in thread